Search code examples
amazon-web-servicesaws-lambdaboto3boto

How can I track the progress/status of an asynchronous AWS Lambda invocation?

I have an API which I use to trigger AWS Lambda jobs. Upon request, the API invokes an AWS Lambda job with InvocationType='Event'. Hereafter, I want to periodically poll if the AWS Lambda job has finished.

The way that would fit best to my architecture, is to store an identifier of the Lambda job in a database and periodically check if the job is finished and what its output is. However, I was not able to find how I can do this.

How can I periodically poll for the result of an AWS Lambda job, and view the output once it has finished?

I have looked into using InvocationType='RequestResponse', but this requires me to store a future, which I cannot do in a database.

Solution

  • There's no built-in way to check for the status of an asynchronous Lambda invocation.

    Asynchronous Lambda invocation, using the event invocation type, is meant to be a fire and forget job. As such, there's no 'progress' or 'status' to get or poll for.

    As you don't want to wait for the Lambda to complete, synchronous Lambda invocation is out of the picture. In this case, you need to write your own logic to keep track of the status.

    One way you could do this is to store a (job) item in a DynamoDB jobs table with 2 attributes:

    1. jobId UUID (String attribute, set as the partition key)
    2. completed boolean flag (Boolean attribute)

    Workflow is then as follows:

    1. Within your API, create & store a new job with completed defaulting to 'false'
    2. Pass the newly-created jobId to the Lambda being invoked in the payload
    3. When the Lambda finishes, lookup the job associated with the passed in jobId within the jobs table & set the completed attribute of the job to true

    You can then periodically poll for the result of the job within the DynamoDB table.

    Or take a look at using DynamoDB Streams as a way to know when a job finishes in near-real time without polling.

    As to viewing the 'output', AWS Lambda just returns a success response without additional information. There is no 'output'. Store any output you might need in persistent storage - maybe an extra output attribute as a String with each job? - & later retrieve it.