The AWS doc states
To allow your function time to process each batch of records, set the source queue's visibility timeout to at least six times the timeout that you configure on your function. The extra time allows for Lambda to retry if your function is throttled while processing a previous batch.
From my understanding, the Lambda polling process will invoke the Lambda function synchronously without any retry, why 6 ? Why not 2 ?
From my understanding, the Lambda polling process will invoke the Lambda function synchronously without any retry, why 6 ? Why not 2 ?
Here's what happens when your Lambda is called through the SQS subscription:
In the past, the only way to throttle the subscription was to set the concurrency on the lambda level. The subscription would always poll for the messages, but it didn't know if it could process them after polling. In this case, if the subscription could not launch another instance of the handler because of the concurrency limits, the message would time out and fail.
These days, you can set maximum concurrency on the subscription level, so that the subscription does not poll for the messages if it sees that it can not process them.
Still, if the lambda is being called by something else, the concurrency issues will still happen.
As you can see, what we've got here is failure to communicate. Lambda, SQS and the subscription don't speak to each other while the message is being processed, and in the past didn't speak even before that.
There are two ways to deal with it:
Put longer timeouts everywhere and hope that with enough retries, the message will eventually get processed. The number 6 was probably chosen because the retry count of 5 is recommended below, and the throttled handler processing the previous message could only time out five times, plus the time it takes to process the new message.
Keep a thread in the lambda processor, extending the message visibility timeout in fixed small increments (say, 5 seconds) while the handler is running. It is not done by default, and requires custom code. Every timeout extension also counts as a request for the billing purposes.