Search code examples
amazon-web-servicesqueueaws-event-bridge

how to manage aws EventBridge queue to execute at most N event at the same time

I have defined an event with Amazon EventBridge that triggers an AWS Lambda function. The Lambda function connects to a MongoDB database and performs a job. At times I receive more than 1000 simultaneous calls, my MongoDB can't handle this number of connections. I need to manage an AWS queue to execute 100 events concurrently while keeping the remaining 900 in the queue, and then proceed to next 100 events, and so on. Is there any solution for this?

I require a FIFO queue for my AWS EventBridge. Instead of executing one event at a time, I need to execute 100 events in parallel.

Solution

  • Overview

    You can change your Lambda target for EventBridge into a SQS FIFO queue. Don't remove your Lambda function since you still need it. Add a SQS trigger for the Lambda function and the queue will be the FIFO queue you created previously. A brief architecture will be as the following: enter image description here

    Key Components

    Yet it's not yet down, you still need to check some attributes for each AWS service for the architecture, or solution, to work smoothly for your scenario based on your requirements. The following are some quick notes:

    1. Create a SQS FIFO queue
      • Visibility timeout: you basically need to set it based on how long a message (data record) that the Lambda function needs to process, for detail, please refer to here.
      • The order of messages leaving a FIFO queue i. Return the oldest message where no other message with the same MessageGroupId is in flight. ii. Return as many messages with the same MessageGroupId as possible. iii. If a message batch is still not full, go back to the first rule. As a result, it’s possible for a single batch to contain messages from multiple MessageGroupIds.
      • Limit throughput: 300 messages/per second
      • Enalbed content-based deduplication
    2. Create a dead-letter queue
      • capture erroneous message
      • capture valid and throttled message

      You can use the dead-letter queue capability to move the messages back to the source queue with just a click of a button on the Amazon SQS console.

    3. Configure your Lambda function properly
      • Reserved concurrency: you need to set it based on the limitaion of batch size for the trigger and your requirement. 100 might be a good start considering the batch size and your requirement.
      • Batch size: For a FIFO queue, the maximum batch size is 10. The actual number of messages in a batch may vary from 1 to 10.
    4. Modification of event processing in the Lambda function.

      Since your Lambda function is now trigged by the SQS FIFO queue, you should expect that the structure of the received event will change.

    Finally, once the solution can work, you should expect to fine-tune the attributes of SQS FIFO queue for lowering throttled events.

    References

    1. Beswick, J. (2019) New for AWS Lambda – SQS FIFO as an event source, AWS Compute Blog. James Beswick. Available at: https://aws.amazon.com/blogs/compute/new-for-aws-lambda-sqs-fifo-as-an-event-source/ (Accessed: April 12, 2023).
    2. Boller, A., Spoon, C., Devlin, B. and Khmelnitsky, M. (2022) Automating notifications from AWS services to Amazon SQS using Amazon EventBridge, Amazon Simple Queue Service Developer Guide. © 2023, Amazon Web Services, Inc. or its affiliates. Available at: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-automating-using-eventbridge.html (Accessed: April 12, 2023).
    3. Boller, A., Spoon, C., Devlin, B. and Khmelnitsky, M. (2022) Configuring a queue to trigger an AWS Lambda function (console), Amazon Simple Queue Service Developer Guide. © 2023, Amazon Web Services, Inc. or its affiliates. Available at: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-configure-lambda-function-trigger.html (Accessed: April 12, 2023).
    4. Boller, A., Spoon, C., Devlin, B., Khmelnitsky, M., Khmelnitsky, M., Khmelnitsky, M. G. and Toledo, R. (2022) Working with Amazon SQS messages, Amazon Simple Queue Service Developer Guide. © 2023, Amazon Web Services, Inc. or its affiliates. Available at: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/working-with-messages.html (Accessed: April 12, 2023).
    5. Collective, M. (2022) How to handle AWS Lambda concurrency with SQS?, Medium. Meta Collective. Available at: https://aws.plainenglish.io/how-to-handle-aws-lambda-concurrency-with-sqs-54a1353cb6fa (Accessed: April 12, 2023).