How do I implement batching with SQS?

Question

I have a lambda (let's call it monitoring lambda) that sends messages to another lambda (notification lambda) via an SQS queue.

Ideally, I want to accomplish the following: Monitoring lambda puts an arbitrary number of notifications onto SQS.

When either one of two conditions occur:

1. a deadline (let's say 30 seconds)
2. the number of notifications in SQS reaches let's say 100 notifications in this case

the notification lambda is triggered with a batch of all of the notifications so that it can aggregate them into a single email.

Is this possible with SQS? Do I need a separate service to poll the SQS queue instead?

Answer 1

When creating a trigger (aka event-source-mapping) for your notification lambda from your SQS queue, you have the option to apply batching behavior. During the creation of an event source mapping, you can specify the number of items to process together (batch size) and the maximum time to wait to group these messages (batch window).

As the lambda function polls the SQS queue, it follows the following pattern: it either waits for the maximum duration specified on the batch window, or in case the number of messages reaches the defined maximum batch size within the batch window duration, it will proceed to process the SQS payload.

The image below illustrates how to set up the Batch Size and Batch Window using the AWS Console:

To know more refer this AWS documentation