AWS - Lambda messages on the queue waiting to be processed are instead getting sent to DLQ queue

Question

My lambda nodejs function: timeout set to 15 minutes with a reserved concurrency of 4.

Messages are fed to it via a standard queue that has an associated DLQ standard queue.

The Lambda trigger config: batch size:1, remaining are set to None or No

The queue config: visibility timeout: 20 minutes, delivery delay:0 seconds, receive message wait time: 20 seconds, message retention period: 2 days, dead-letter queue is set to 3 Maximum receives

The dlq queue config visibility timeout: 30 minutes, delivery delay:0 seconds, receive message wait time: 0 seconds, message retention period: 4 day

Observations

1. Some of the messages are taking in excess of 15mins to be processed by Lambda and as expected are timing out.
2. Because of the above, I would expect to see the remaining messages waiting in the queue for their turn to be processed, instead they are getting routed to the DLQ, lambda logs indicate no attempt to process these messages.
3. In the DLQ queue, when I poll the messages, I can see the receive count incrementing, it was at 7 when I last looked.
4. When I looked at the DLQ queue, I can see that the messages available:0, messages in flight: 57

I have misunderstood how to set this up, can anyone explain what is happening here? ideally I would want the messages to wait for their turn.

Answer 1

Since you have set the concurrency count to 4, it means that only four lambda instances are available to process your SQS messages. When an SQS message appears in the queue, it needs a consumer (a lambda function) to pick it up for processing. However, because your lambda function takes around 15 minutes to complete, sometimes there may not be enough consumers available to handle the SQS messages. As a result, some of the SQS messages may not be processed by a lambda function and end up in the dead letter queue.

To address this issue, you have a couple of options. One option is to increase the concurrency limit of your lambda function. By doing so, more messages can be processed simultaneously. If you're unsure about the required concurrency limit, you can allow the lambda function to auto-scale based on the workload.

Alternatively, you can keep the concurrency limit on the lambda function as it is and instead increase the Maximum receives count from 3 to some higher limit.

Both of these approaches will definitely help to resolve the problem that you mentioned.