Search code examples
amazon-web-servicesaws-lambdaamazon-sqsaws-msk

How to limit max concurrency for SQS queue receivers based on each type of message?


I lambda function that returns a bunch of requests that I want to send to an external service. This external service has an api with the following rate limit:

Paramters: 
type: enum
content: list of n elements

Rate limit: 
For each type, there can be no more than 3 concurrent slots. Each slot will be will need to wait n milliseconds between requests. 

I want to throttle the requests being sent. My deign is to use sqs to queue the request payloads and set an autoscaling lambda to consume them. The sqs configuration includes a max_concurrency, but I dont see a way to configure that based on the message content.

Is there a way to implement the rate limit throttling described above using SQS? Or should I use another solution like Amazon MSK?

Example:

For below queue, I want a lambda function to be able to concurrently process 3 messages for type 1, 3 messages for type 2, and so on.

queue: 
type=1, payload
type=1, payload
type=1, payload
type=1, payload
type=2, payload
type=2, payload
type=2, payload
type=3, payload

Solution

  • If you want to process messages differently, you would need to: