I am using AWS MSK(Managed Streaming Service for Apache Kafka) as a trigger to lambda function. I have a window of zero seconds and batch size of one.
I have observed that the invocation of lambda function from lambda service after polling message from MSK, is synchronous; meaning it does not commit the message back to the partition as long as the lambda execution is not successful. In configurations for lambda, under asynchronous invocation, I have configured the message expiration to be 6 hours and retries as 2. What have I observed is that, though the lambda is retried as long as the message is not committed back to the topic partition as long as the lambda execution successful, but even after 6 hour, I could not see any message in the set dead letter queue.
What happens when the message which is polled at the first time, why does the lambda service not send this message to the configured SQS DLQ after the configured message expiration time in asynchronous invocation section?
In the AWS documents here it says the lambda retries as long as the message is not processed successfully or it expires.
Does that mean MSK trigger for lambda is fully Synchronous and will never implicitly send the message to the dead letter queue or drop it after expiration if queue is not configured?
So after playing around I had figured out, it is completely synchronous operation. How does it work is like, as long as the lambda execution is not successful and successfully completed, i.e. without any errors, the current offset will not be committed. Also there will not be any concurrent operations beyond this offset for this partition. AWS lambda's internal service takes care of it. Anything from kafka is a poll based operation, rather a push. So an internal service called event source mapping, polls events from the partitions based on the batch size and batch number you have provided in you configurations. And then it invokes the lambda function. It waits for the successful completion of the the execution and once lambda is executed without errors, it commits this offset and polls the other one and invokes the lambda.
There is no way that I could figure out, where you can manually commit the offset using lambda code. Instead, if you want that few particular errors should not hamper your pipelines and offsets should keep coming in, you need to smartly handle that inside your exceptions. If the exception block does not raises/ or your code does not raises any runtime/errors, the execution would be successful and offset would be committed and you will keep receiving new offsets. But design your application smartly, failing the lambda is also important as long as you don't know the error or don't know what are you going to do with that.
Also, the messages this way does not expire and event source mapping keeps polling the same message again and again, this nothing is written to the dead letter queue.
Hope this helps, feel free to write further queries/questions/suggestions in comment.