Search code examples
amazon-web-servicesamazon-efsaws-backup

AWS Backup jobs for EFS file systems are empty

I have a backup plan:

{
    "BackupPlan": {
        "BackupPlanName": "prometheus_dev",
        "Rules": [
            {
                "RuleName": "prometheus_dev",
                "TargetBackupVaultName": "prometheus_eu_central_1_dev",
                "ScheduleExpression": "cron(15 * ? * * *)",
                "StartWindowMinutes": 60,
                "CompletionWindowMinutes": 180,
                "Lifecycle": {
                    "DeleteAfterDays": 7
                },
                "RuleId": "ffcd7e8c-9b14-4e2b-89f0-d8cbe5b5ae25",
                "CopyActions": [
                    {
                        "Lifecycle": {
                            "DeleteAfterDays": 7
                        },
                        "DestinationBackupVaultArn": "arn:aws:backup:eu-west-1:614797193252:backup-vault:prometheus_backup_eu_west_1_dev"
                    }
                ],
                "EnableContinuousBackup": false
            }
        ]
    },
    "BackupPlanId": "830e8bb0-e3c6-4a96-8fe9-45a3ab97de40",
    "BackupPlanArn": "arn:aws:backup:eu-central-1:614797193252:backup-plan:830e8bb0-e3c6-4a96-8fe9-45a3ab97de40",
    "VersionId": "YzUxMjhkODgtZTc0MC00NDA1LWEwYzktNmE4NDFhMDE5MTA4",
    "CreationDate": "2023-10-13T13:42:22.048000+02:00",
    "LastExecutionDate": "2023-10-20T11:16:48.451000+02:00"
}

The file system is protected:

{
    "ResourceArn": "arn:aws:elasticfilesystem:eu-central-1:614797193252:file-system/fs-0318e5506f10caf1e",
    "ResourceType": "EFS",
    "LastBackupTime": "2023-10-20T11:22:48.159000+02:00",
    "ResourceName": "prometheus_dev"
}

And backups are taken hourly:

// ...
{
            "AccountId": "614797193252",
            "BackupJobId": "06178992-5DF7-17BA-1E38-C969357B644A",
            "BackupVaultName": "prometheus_eu_central_1_dev",
            "BackupVaultArn": "arn:aws:backup:eu-central-1:614797193252:backup-vault:prometheus_eu_central_1_dev",
            "RecoveryPointArn": "arn:aws:backup:eu-central-1:614797193252:recovery-point:58cb7957-5b0c-4038-a579-af78aadbc506",
            "ResourceArn": "arn:aws:elasticfilesystem:eu-central-1:614797193252:file-system/fs-0318e5506f10caf1e",
            "CreationDate": "2023-10-20T08:15:00+02:00",
            "CompletionDate": "2023-10-20T08:25:07.003000+02:00",
            "State": "COMPLETED",
            "PercentDone": "100.0",
            "BackupSizeInBytes": 0,
            "IamRoleArn": "arn:aws:iam::614797193252:role/prometheus_backup_dev",
            "CreatedBy": {
                "BackupPlanId": "830e8bb0-e3c6-4a96-8fe9-45a3ab97de40",
                "BackupPlanArn": "arn:aws:backup:eu-central-1:614797193252:backup-plan:830e8bb0-e3c6-4a96-8fe9-45a3ab97de40",
                "BackupPlanVersion": "YzUxMjhkODgtZTc0MC00NDA1LWEwYzktNmE4NDFhMDE5MTA4",
                "BackupRuleId": "ffcd7e8c-9b14-4e2b-89f0-d8cbe5b5ae25"
            },
            "StartBy": "2023-10-20T09:15:00+02:00",
            "ResourceType": "EFS",
            "IsParent": false,
            "ResourceName": "prometheus_dev"
        },
        {
            "AccountId": "614797193252",
            "BackupJobId": "DE456E1F-1A49-4A36-507A-32646DD0AE85",
            "BackupVaultName": "prometheus_eu_central_1_dev",
            "BackupVaultArn": "arn:aws:backup:eu-central-1:614797193252:backup-vault:prometheus_eu_central_1_dev",
            "RecoveryPointArn": "arn:aws:backup:eu-central-1:614797193252:recovery-point:998837d7-e0c8-4505-9d4f-ff19ca1f69c7",
            "ResourceArn": "arn:aws:elasticfilesystem:eu-central-1:614797193252:file-system/fs-0318e5506f10caf1e",
            "CreationDate": "2023-10-20T07:15:00+02:00",
            "CompletionDate": "2023-10-20T07:25:52.381000+02:00",
            "State": "COMPLETED",
            "PercentDone": "100.0",
            "BackupSizeInBytes": 0,
            "IamRoleArn": "arn:aws:iam::614797193252:role/prometheus_backup_dev",
            "CreatedBy": {
                "BackupPlanId": "830e8bb0-e3c6-4a96-8fe9-45a3ab97de40",
                "BackupPlanArn": "arn:aws:backup:eu-central-1:614797193252:backup-plan:830e8bb0-e3c6-4a96-8fe9-45a3ab97de40",
                "BackupPlanVersion": "YzUxMjhkODgtZTc0MC00NDA1LWEwYzktNmE4NDFhMDE5MTA4",
                "BackupRuleId": "ffcd7e8c-9b14-4e2b-89f0-d8cbe5b5ae25"
            },
            "StartBy": "2023-10-20T08:15:00+02:00",
            "ResourceType": "EFS",
            "IsParent": false,
            "ResourceName": "prometheus_dev"
        },
// ...

The problem is that ALL recovery points are empty, even if the file system is not:

{
// ...
            "SizeInBytes": {
                "Value": 67465216,
                "Timestamp": "2023-10-20T10:41:44+02:00",
                "ValueInIA": 0,
                "ValueInStandard": 67465216
            }
// ...
}

I've also tried to start a backup job:

aws backup start-backup-job \
    --backup-vault-name prometheus_eu_central_1_dev \
    --resource-arn arn:aws:elasticfilesystem:eu-central-1:614797193252:file-system/fs-0318e5506f10caf1e \
    --iam-role-arn arn:aws:iam::614797193252:role/prometheus_backup_dev

Which produced the following response:

{
    "BackupJobId": "7a010939-11cd-4d6f-bf2e-bbec0fc50452",
    "RecoveryPointArn": "arn:aws:backup:eu-central-1:614797193252:recovery-point:ac338f60-9a4e-4c24-817e-2e0e15a72d03",
    "CreationDate": "2023-10-20T11:22:48.159000+02:00",
    "IsParent": false
}

But again the backup is empty:

{
    "AccountId": "614797193252",
    "BackupJobId": "7a010939-11cd-4d6f-bf2e-bbec0fc50452",
    "BackupVaultName": "prometheus_eu_central_1_dev",
    "BackupVaultArn": "arn:aws:backup:eu-central-1:614797193252:backup-vault:prometheus_eu_central_1_dev",
    "RecoveryPointArn": "arn:aws:backup:eu-central-1:614797193252:recovery-point:ac338f60-9a4e-4c24-817e-2e0e15a72d03",
    "ResourceArn": "arn:aws:elasticfilesystem:eu-central-1:614797193252:file-system/fs-0318e5506f10caf1e",
    "CreationDate": "2023-10-20T11:22:48.159000+02:00",
    "CompletionDate": "2023-10-20T11:22:55.105000+02:00",
    "State": "COMPLETED",
    "PercentDone": "100.0",
    "BackupSizeInBytes": 0,
    "IamRoleArn": "arn:aws:iam::614797193252:role/prometheus_backup_dev",
    "ResourceType": "EFS",
    "BytesTransferred": 0,
    "StartBy": "2023-10-20T19:22:48.159000+02:00",
    "IsParent": false,
    "ResourceName": "prometheus_dev"
}

Why are these backups empty? How can i further debug this?

I see no errors anywhere, and the role used has the necessary permissions, since it's using AWS managed policy.

EDIT:

  • After @Tsal Troser, I noticed I forgot to mention that the reason I discovered that EFS backups, for some reason, are not working, is exactly because I tried to use a recovery point to create a new EFS. The EFS is created, but it is empty.
  • The same can't be said for RDS recovery points. Although describing a backup also shows a BackupSizeInBytes as 0, creating an instance from the recovery point provisions an instance with data as expected.

EDIT 2:

Solution

  • Because AWS Backup perform incremental backups. The initial backup will be a full backup then the following, even the on-demand backup, will be an incremental backup.

    AWS Backup performs incremental backups of EFS file systems. During the initial backup, a copy of the entire file system is made. During subsequent backups of that file system, only files and directories that have been changed, added, or removed are copied. https://docs.aws.amazon.com/efs/latest/ug/awsbackup.html#incremental-backup

    "BackupSizeInBytes": 0,

    This just mean that from the last recovery point/backup to the next, there wasn't any change.

    I checked my backups as well and I got the same. But that's because I'm not actively using my EFS.

    {
> aws backup describe-recovery-point --backup-vault-name myEfsVault --recovery-point-arn <recoveryPointArn>
    ...
    "Status": "COMPLETED",
    "CreationDate": "2023-10-20T07:00:00+02:00",
    "CompletionDate": "2023-10-20T09:15:48.157000+02:00",
    "BackupSizeInBytes": 0,
    ...
}

    Here's what you can do to confirm your backup integrity:

    • Perform a full restore and check if your files are there. You can cross check with the original.
    • Upload a large file in your EFS then perform an on-demand backup then check the backup job/recovery point details.

    Bonus

    Here's an AWS recommendation for performing data recovery validation with AWS Backup.

    https://aws.amazon.com/blogs/storage/automate-data-recovery-validation-with-aws-backup/

    Edit

    Backup testing

    New Initial Backup

    }
    ...
    "Status": "COMPLETED",
    "CreationDate": "2023-10-20T14:54:57.001000+02:00",
    "CompletionDate": "2023-10-20T14:55:07.614000+02:00",
    "BackupSizeInBytes": 1199258039,
    ...
}

    Followup on-demand backup:

    {
    ...
    "Status": "COMPLETED",
    "CreationDate": "2023-10-20T14:57:43.050000+02:00",
    "CompletionDate": "2023-10-20T14:57:50.314000+02:00",
    "BackupSizeInBytes": 0,
    ...
}

    Restore test:

    Restore job: enter image description here

    Restored EFS and Original EFS (Original names redacted): enter image description here

    Create Mount Targets (Select subnet, SG, AZ) enter image description here

    Mount restored FS and check contents:

    [root@test-server ~]# mkdir /efs-restore
[root@test-server ~]# mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport,mountport=2049 10.10.10.113:/ /efs-restore

[root@test-server efs-restore]# df -h /efs-restore
Filesystem      Size  Used Avail Use% Mounted on
10.10.10.113:/  8.0E     0  8.0E   0% /efs-restore

[root@test-server efs-restore]# ll /efs-restore
drwxrwxr-x 4 root root 6144 Aug 14 12:39 aws-backup-restore_2023-10-20T13-00-49-858648977Z

[root@test-server efs-restore]# ll /efs-restore/aws-backup-restore_2023-10-20T13-00-49-858648977Z/
drw--w---- 2 root     root     6144 Oct 20 15:00 aws-backup-lost+found_2023-10-20T13-00-32-067742883Z
drwxrwxr-x 3 ec2-user ec2-user 6144 Aug  1 11:38 logs  # <-- my application logs

[root@test-server efs-restore]# ll /efs-restore/aws-backup-restore_2023-10-20T13-00-49-858648977Z/logs/path/to/my/application/
total 100
drwxr-xr-x 3 ec2-user ec2-user 6144 Oct 12 14:55 ip-10-10-4-123
drwxr-xr-x 3 ec2-user ec2-user 6144 Oct 12 17:55 ip-10-10-4-127
drwxrwxr-x 3 ec2-user ec2-user 6144 Sep 11 16:50 ip-10-10-4-151
drwxr-xr-x 3 ec2-user ec2-user 6144 Oct 11 09:28 ip-10-10-4-160

    Restore test 2:

    Deleted the initial backup full backup to simulate the retention period. Then performed another restore, everything is still there.