Search code examples
pythonaws-lambdaterraformamazon-sqs

Lambda SQS Terraform Concurrency

My number of Lambda executions aren't matching the number of SQS messages. I want 1 lambda execution for 1 message on my SQS queue.

My SQS queue has 83 messages to be processed, but my cloudwatch logs only have 44 logs in the log group when all the processing is finished. Why are half of my messages seeming to get lost? The lambda's are nto timing out and the ones that are executing are completing their processing way ahead of the timeout and processing only one message. Are there more settings I have to figure out?

My terraform code:

resource "aws_lambda_event_source_mapping" "event_source_mapping" {
  batch_size        = 1
  event_source_arn  = aws_sqs_queue.my_queue.arn
  enabled           = true
  function_name     = aws_lambda_function.my_lambda.arn
}

My Python code (batch_messages should always just be 1) in my Lambda:

    queue = sqs.get_queue_by_name(QueueName='my-queue')
    batch_messages = queue.receive_messages(MaxNumberOfMessages=1)

    #Once done processing
    receipt_handle = tickers.receipt_handle 
    sqs_client.delete_message(QueueUrl=queue_url,ReceiptHandle=receipt_handle)
Solution

  • You have an event source mapping, so Lambda will automatically poll SQS for messages and send them into your Lambda function, inside the handler's event object.

    Inside your Lambda function's code, you are also calling queue.receive_messages() which is doing additional polling to retrieve another message from the queue. Your code is completely ignoring the message retrieved via the event source mapping, and doing manual message retrieval. That means each time the Lambda function is called one message is removed from the queue (by the event source mapping) which you are throwing away, and another message is removed from the queue (via your code) and processed.