Search code examples
node.jsemailaws-sdkamazon-sesrate-limiting

Sending emails by SES as fast as possible without exceeding the rate limit

I'm currently using SendGrid to send emails from many serverless node.js Lambda functions, and planning to transfer from SendGrid to SES (because of the reasonable price).

SendGrid's rate limit is so high that sending emails to users using for-loops by some lambda functions concurrently never caused a problem. But SES has lower rate limit (50 emails per second in my case), and maximum sending rate limit exceeded error occurs.

How is it possible to send emails as fast as possible without exceeding the rate limit?

My current plan is to send email request to SQS queue by many lambda functions, and receive SQS message to send emails by 1 Lambda function running without stopping time. But I'm not sure how to control the speed of sending requests to SES.

Solution

  • My conclusion

    I decided to create two SQS queues (A and B). Queue B is for triggering an email sending function, and queue A is for adding messages in the waiting queue. So I only need to control the speed of moving messages from queue A to B, and that makes everything simple.

    enter image description here

    I considered the following things.

    Error handling

    At first, I was thinking of using while-loops to send many emails by a single Lambda function, but that makes error handling difficult. When the rate limit exceeded error or some other unexpected exception occurs, it's necessary to wait for a while and retry, but it could exceed the timeout of Lambda function.

    Scalability

    Receiving 50 messages from SQS queue and sending 50 requests to SES in 1 second by a single Lambda function could be possible, but if the rate limit goes up to 300 or 500, it might be impossible to handle that many requests in a second. Moving messages from one SQS queue to another, in order to invoke a Lambda function per email, is relatively easy to scale.

    Email priority

    Signup confirmation email with a verification code needs to be sent instantly, but campaign emails can be sent on a delay. With two SQS queues, I can control the priority.

    AWS cost

    Invoking a Lambda function for each email takes more time than sending many emails in a single Lambda function, but the AWS cost for that is not very expensive. If it takes 1 second to send a request to SES, it costs 0.000002083 USD for Lambda execution with 128 MB memory (N.Virginia region). That's only 0.02083 USD for 1000 emails and can be ignored.