How do I speed up the rate at which DynamoDB streams fire INSERT events?

Question

I have a setup where every time an item is inserted in my table, I call a Lambda function. I'm trying to cut huge images in 256x256 tiles, it's a one time job but I need a few million lambda calls.

The system works, but very slowly, around 30 invocations/sec. I can see the iterator age increasing in the lambda monitor.

• The lambda concurrency limit is 1000
• The DynamoDB is using provisioned capacity, auto-scaling, 1k reads/writes max

Where is the bottleneck coming from?

Answer 1

First thing to ensure is that you have retries enabled and not using the default -1 value which can cause your stream shard to block.

Then you can increase the performance doing a mixture of the following:

1. Increase your Lambda memory
2. Increase the parallelization factor of the Lambda (up to 10X)
3. Pre-warm your DynamoDB table to give you more shards
4. Reduce BatchSize
5. Refine your Lambda logic to be more efficient.