AWS SQS based trigger for Lambda drops messages

Question

I have a SQS queue configured with a Dead Letter Queue. There are 2 Lambdas that are trigger by the SQS queue events. Each of them have a different filter which enables them to process messages meant for them. The filters look like below:

{ "body" : { "ErpId" : [ "myErp1" ] } }

and

{ "body" : { "ErpId" : [ "someOtherErp" ] } } 

When I send 6 messages to the SQS queue where 3 have ErpId myErpId and the other 3 have ErpId someOtherErp, it is observed that some or all of the messages for myErpId are dropped.

I can see all 6 of them come into SQS queue, 3 of them are picked up by Lambda for someOtherErp but the other Lambda only receives some of the messages. Those messages are not found in the SQS queue or in the Dead Letter Queue and neither can I see them in the CloudWatch Log Group for either Lambdas.

Any idea as to what is going on here?

Update: both the Lambdas drop few messages (or the trigger drops them)

Answer 1

I ran into that problem last year and wrote a blog about it (yes, yes - written by me and relevant).

If you have a queue with multiple subscribers, e.g. Lambda & EC2 where the Lambda has a filter, you may lose some or all of the messages that don't match the filter because Lambda discards the messages that don't.

A workaround would be an SNS topic with a queue per Lambda attached to it to duplicate the messages.