Search code examples
amazon-web-servicesaws-lambdaamazon-sqsdead-letter

Re-process DLQ events in Lambda


I have an AWS Lambda Function 'A' with a SQS DeadLetterQueue configured. When the Lambda fails to process an event, this is correctly sent to the DLQ. Is there a way to re-process events that ended into a DLQ?

I found two solution, but they both have drawbacks:

  1. Create a new Lambda Function 'B' that reads from the SQS and then sends the events one by one to the previous Lambda 'A'. -> Here I have to write new code and deploy a new Function
  2. Trigger again Lambda 'A' just when an event arrives in the SQS -> This looks dangerous as I can incur in looping executions

My ideal solution should be re-processing on demand the discarded events with Lambda 'A', without creating a new Lambda 'B' from scratch. Is there a way to accomplish this?


Solution

  • Finally, I didn't find any solution from AWS to reprocess the DLQ events of a Lambda Function. Then I created my own custom Lambda Function (I hope that this will be helpful to other developers with same issue):

    import boto3
    
    lamb = boto3.client('lambda')
    sqs = boto3.resource('sqs')
    queue = sqs.get_queue_by_name(QueueName='my_dlq_name')
    
    
    def lambda_handler(event, context):
        for _ in range(100):
            messages_to_delete = []
            for message in queue.receive_messages(MaxNumberOfMessages=10):
                payload_bytes_array = bytes(message.body, encoding='utf8')
                # print(payload_bytes_array)
                lamb.invoke(
                    FunctionName='my_lambda_name',
                    InvocationType="Event",  # Event = Invoke the function asynchronously.
                    Payload=payload_bytes_array
                )
    
                # Add message to delete
                messages_to_delete.append({
                    'Id': message.message_id,
                    'ReceiptHandle': message.receipt_handle
                })
    
            # If you don't receive any notifications the messages_to_delete list will be empty
            if len(messages_to_delete) == 0:
                break
            # Delete messages to remove them from SQS queue handle any errors
            else:
                deleted = queue.delete_messages(Entries=messages_to_delete)
                print(deleted)
    

    Part of the code is inspired by this post