Search code examples
postgresqlpsycopg2database-backupswal-e

Postgresql: How to take incremental backup with wal-e

I am new to postgresql. I've connected to my demo database with psycopg2 and inserted data with python faker. I want to take incremental backup with the tool WAL-E and want to see how it actually works. But the few tutorials aren't helping that much as they're not for naive users.

Can someone help explaining the steps simply how to backup data with WAL-E? That would be a great help ! I'm using psql 10.4 in Ubuntu 18.04.

Thanks

Solution

  • Install Python & Dependencies

    apt-get install python-dev python-virtualenv libevent-dev pv lzop daemontools
sudo apt-get install -y python3-pip
sudo apt-get install build-essential libssl-dev libffi-dev
sudo apt-get install -y python3-venv

    Create & Activate Virtual Environment

    Sudo python3 -m venv /opt/wal-e
source /opt/wal-e/bin/activate

    Install Wal-e

    sudo /opt/wal-e/bin/pip install wal-e[aws]

    Setup Environment Variables

    umask u=rwx,g=rx,o=
mkdir -p /etc/wal-e.d/env
sudo -i -u root
echo "Your AWS Secret Key" > /etc/wal-e.d/env/AWS_SECRET_ACCESS_KEY
echo "Your AWS Access Key" > /etc/wal-e.d/env/AWS_ACCESS_KEY_ID
echo 'Your AWS Region > /etc/wal-e.d/env/AWS_REGION
echo 's3://some-bucket/directory/or/whatever' > /etc/wal-e.d/env/WALE_S3_PREFIX
sudo chown -R root:postgres /etc/wal-e.d

    Change Postgres Config to Activate Archiving

    sudo nano /etc/postgresql/10/main/postgresql.conf
wal_level = replica
archive_mode = on
archive_command = '/usr/bin/envdir /etc/wal-e.d/env /opt/wal-e/bin/wal-e wal-push %p'
archive_timeout = 60
sudo systemctl restart postgresql

    Test Wal-e is Working

    sudo -i -u postgres
/usr/bin/envdir /etc/wal-e.d/env /opt/wal-e/bin/wal-e backup-list

    Now Push Your First Base Backup to S3

    /usr/bin/envdir /etc/wal-e.d/env /opt/wal-e/bin/wal-e backup-push /var/lib/postgresql/10/main

    Now Set the Cron For Regular Backup

    sudo -i -u postgres 

mkdir logs
touch backup.log
touch deletebackup.log

crontab -e

00 00 * * * /usr/bin/envdir /etc/wal-e.d/env /opt/wal-e/bin/wal-e backup-push /var/lib/postgresql/10/main > ~/logs/backup.log 2>&1
00 00 * * * /usr/bin/envdir /etc/wal-e.d/env  /opt/wal-e/bin/wal-e delete --confirm retain 1 > ~/logs/deletebackup.log 2>&1