Search code examples
backupduplicity

How to use duplicity for a rotating backup schedule

I'm trying to use duplicity to create a rotating backup schedule: hourly backups for the last day, daily backups for the last week, weekly backups for the last month, monthly backups forever (to S3).

If there's a better way to accomplish this than what I'm about to describe, I'll take it. But I'll also take figuring out why this isn't working.

I've created a set of scripts: hourly, hourly_clean, daily, etc...., and then put them in cron

 12  *  *  *  * root /etc/duplicity_backup/hourly
 16  *  *  *  * root /etc/duplicity_backup/hourly_clean
 18  2  *  *  * root /etc/duplicity_backup/daily
 22  2  *  *  * root /etc/duplicity_backup/daily_clean
 24  3  *  *  7 root /etc/duplicity_backup/weekly
 28  3  *  *  7 root /etc/duplicity_backup/weekly_clean
 30  4  28 *  * root /etc/duplicity_backup/monthly

Here are the hourly and daily scripts:

#!/usr/bin/ksh
#
# hourly
#
AWS_ACCESS_KEY_ID=<my_key_id>;
AWS_SECRET_ACCESS_KEY=<my_key>;
PASSPHRASE=foo;

export AWS_ACCESS_KEY_ID;
export AWS_SECRET_ACCESS_KEY;
export PASSPHRASE;

/usr/local/bin/duplicity --archive-dir /mnt/data/var/cache/duplicity \
    --name webserver_vhosts --s3-region-name eu-central-1 \
    --tempdir /mnt/data/var/tmp/ \
    --full-if-older-than 1D \
    --include **/wp-content \
    --exclude **/wp-content/themes --exclude **/wp-content/plugins \
    /mnt/data/www/vhosts/ boto3+s3://mirovoy-backup-webserver/hourly;

#!/usr/bin/ksh
#
# daily
#
AWS_ACCESS_KEY_ID=<my_key_id>;
AWS_SECRET_ACCESS_KEY=<my_key>;
PASSPHRASE=foo;

export AWS_ACCESS_KEY_ID;
export AWS_SECRET_ACCESS_KEY;
export PASSPHRASE;

/usr/local/bin/duplicity full --archive-dir /mnt/data/var/cache/duplicity \
    --dry-run --name webserver_vhosts --s3-region-name eu-central-1 \
    --tempdir /mnt/data/var/tmp/ \
    --full-if-older-than 1W \
    --s3-use-ia \
    --include **/wp-content \
    --exclude **/wp-content/themes --exclude **/wp-content/plugins \
    /mnt/data/www/vhosts/ boto3+s3://mirovoy-backup-webserver/daily;

The hourly backups work fine, but the others, AFAICT, don't do anything.

If I try and run them manually, it gives a warning, but no errors:

$sudo ./daily
Warning, found signatures but no corresponding backup files
Sync would remove the following spurious local files:
duplicity-full-signatures.20210317T001201Z.sigtar.gz
duplicity-full-signatures.20210319T011202Z.sigtar.gz
duplicity-full.20210317T001201Z.manifest
duplicity-full.20210319T011202Z.manifest
duplicity-inc.20210317T001201Z.to.20210317T011201Z.manifest
duplicity-inc.20210317T011201Z.to.20210317T021201Z.manifest
[... lots more local files]
Last full backup date: none
--------------[ Backup Statistics ]--------------
StartTime 1616136196.38 (Fri Mar 19 06:43:16 2021)
EndTime 1616136207.36 (Fri Mar 19 06:43:27 2021)
ElapsedTime 10.98 (10.98 seconds)
SourceFiles 97158
SourceFileSize 1083141759 (1.01 GB)
NewFiles 54089
NewFileSize 1083141759 (1.01 GB)
DeletedFiles 0
ChangedFiles 0
ChangedFileSize 0 (0 bytes)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 54089
RawDeltaSize 0 (0 bytes)
TotalDestinationSizeChange 0 (0 bytes)
Errors 0
-------------------------------------------------

As I said, the hourly works fine, but the others aren't pushing anything to S3. Why are the daily/weekly/monthly backups not working? Alternatively, is there a better way to get duplicity to accomplish this?

Solution

  • The daily script has the option "--dry-run", which according to the duplicity man page has duplicity "Calculate what would be done, but do not perform any backend actions". The hourly script omits that option.