Calling AWS RDS CreateDBSnapshotAsync Asynchronously "Set It And Forget It"

Question

In an AWS Lambda function, I would like to be able to call a component to create a RDS DB Snapshot. There is an async method on the client named CreateDBSnapshotAsync. But, because this is AWS Lambda, I only have 5 minutes to complete the task. So, if I await it, the AWS Lambda function will timeout. And, apparently when it times out, the call is cancelled and then the snapshot is not completed.

Is there some way I can make the call in a COMPLETELY asynchronously way so that once I invoke it, it will complete no matter if my Lambda function times out or not? In other words, I don't care about the result, I just want to invoke the process and move on, a "set it and forget it" mentality.

My call (without the await, obviously) is as below

        using (var rdsClient = new AmazonRDSClient())
        {
            Task<CreateDBSnapshotResponse> response = rdsClient.CreateDBSnapshotAsync(new CreateDBSnapshotRequest($"MySnapShot", instanceId));
        }

As requested, here's the full method:

    public async Task<CloudFormationResponse> MigrateDatabase(CloudFormationRequest request, ILambdaContext context)
    {
        LambdaLogger.Log($"{nameof(MigrateDatabase)} invoked: " + JsonConvert.SerializeObject(request));


        if (request.RequestType != "Delete")
        {
            try
            {
                var migrations = this.Context.Database.GetPendingMigrations().OrderBy(b=>b).ToList();
                for (int i = 0; i < migrations.Count(); i++)
                {
                    string thisMigration = migrations [i];
                    this.ApplyMigrationInternal(thisMigration);
                }
                this.TakeSnapshotAsync(context,migrations.Last());
                return await CloudFormationResponse.CompleteCloudFormationResponse(null, request, context);

            }
            catch (Exception e)
            {
                LambdaLogger.Log(e.ToString());
                if (e.InnerException != null) LambdaLogger.Log(e.InnerException.ToString());
                return await CloudFormationResponse.CompleteCloudFormationResponse(e, request, context);
            }
        }
        return await CloudFormationResponse.CompleteCloudFormationResponse(null, request, context);

    }

    internal void TakeSnapshotAsync(ILambdaContext context, string migration)
    {

        var instanceId = this.GetEnvironmentVariable(nameof(DBInstance));

        using (var rdsClient = new AmazonRDSClient())
        {


            Task<CreateDBSnapshotResponse> response = rdsClient.CreateDBSnapshotAsync(new CreateDBSnapshotRequest($"{instanceId}{migration.Replace('_','-')}", instanceId));
            while (context.RemainingTime > TimeSpan.FromSeconds(15))
            {
                Thread.Sleep(15000);
            }
        }
    }

Answer 1

First refactor that sub function to use proper async syntax along with the use of Task.WhenAny.

internal async Task TakeSnapshotAsync(ILambdaContext context, string migration) {
    var instanceId = this.GetEnvironmentVariable(nameof(DBInstance));
    //don't wrap in using block or it will be disposed before you are done with it.
    var rdsClient = new AmazonRDSClient();
    var request = new CreateDBSnapshotRequest($"{instanceId}{migration.Replace('_','-')}", instanceId);
    //don't await this long running task
    Task<CreateDBSnapshotResponse> response = rdsClient.CreateDBSnapshotAsync(request);
    Task delay = Task.Run(async () => {
        while (context.RemainingTime > TimeSpan.FromSeconds(15)) {
            await Task.Delay(15000); //Don't mix Thread.Sleep. use Task.Delay and await it.
        }
    }
    // The call returns as soon as the first operation completes, 
    // even if the others are still running.
    await Task.WhenAny(response, delay);
}

So if the RemainingTime runs out, it will break out of the call even if the snap shot task is still running so that the request does not time out.

Now you should be able to await the snapshot while there is still time available in the context

public async Task<CloudFormationResponse> MigrateDatabase(CloudFormationRequest request, ILambdaContext context) {
    LambdaLogger.Log($"{nameof(MigrateDatabase)} invoked: " + JsonConvert.SerializeObject(request));

    if (request.RequestType != "Delete") {
        try {
            var migrations = this.Context.Database.GetPendingMigrations().OrderBy(b=>b).ToList();
            for (int i = 0; i < migrations.Count(); i++) {
                string thisMigration = migrations [i];
                this.ApplyMigrationInternal(thisMigration);
            }
            await this.TakeSnapshotAsync(context, migrations.Last());
            return await CloudFormationResponse.CompleteCloudFormationResponse(null, request, context);
        } catch (Exception e) {
            LambdaLogger.Log(e.ToString());
            if (e.InnerException != null) LambdaLogger.Log(e.InnerException.ToString());
            return await CloudFormationResponse.CompleteCloudFormationResponse(e, request, context);
        }
    }
    return await CloudFormationResponse.CompleteCloudFormationResponse(null, request, context);

}

This should also allow for any exceptions thrown by the RDS client to be caught by the currently executing thread. Which should help with troubleshooting any exception messages.

Some interesting information from documentation.

Using Async in C# Functions with AWS Lambda

If you know your Lambda function will require a long-running process, such as uploading large files to Amazon S3 or reading a large stream of records from DynamoDB, you can take advantage of the async/await pattern. When you use this signature, Lambda executes the function synchronously and waits for the function to return a response or for execution to time out.

From docs about timeouts

Function Settings

...

• Timeout – The amount of time that Lambda allows a function to run before stopping it. The default is 3 seconds. The maximum allowed value is 900 seconds.

If getting a HTTP timeout then shorten the delay but leave the long running task. You still use the Task.WhenAny to give the long running task an opportunity to finish first even if that is not the expectation.

internal async Task TakeSnapshotAsync(ILambdaContext context, string migration) {
    var instanceId = this.GetEnvironmentVariable(nameof(DBInstance));
    //don't wrap in using block or it will be disposed before you are done with it.
    var rdsClient = new AmazonRDSClient();
    var request = new CreateDBSnapshotRequest($"{instanceId}{migration.Replace('_','-')}", instanceId);
    //don't await this long running task
    Task<CreateDBSnapshotResponse> response = rdsClient.CreateDBSnapshotAsync(request);
    Task delay = Task.Delay(TimeSpan.FromSeconds(2.5));
    // The call returns as soon as the first operation completes, 
    // even if the others are still running.
    await Task.WhenAny(response, delay);
}