How can i aggregate data from multiple lambdas in aws

Question

I have SNS Topic which triggers 50 Lambdas in Multiple Accounts

Now each lambda produces some output in json format.

I want to aggregate all those individual json into one list and then pass that into Another SNS Topic

whats the best to achieve to aggregate data

Answer 1

There are a couple of architecture solutions you can use to solve this. There is probably not a "right one", it will depends on the volume of data, frequency of triggers and budget.

You will need some shared storage where your 50 lambdas functions can temporary store their results, and another component, most probably another lambda function in charge of the aggregation to produce the final result.

Depending on the volume of data to handle, I would first consider a shared Amazon S3 bucket where all your 50 functions can drop their piece of JSON, and the aggregation function could read and assemble all the pieces. Other services that can act as a shared storage are Amazon DynamoDB and Amazon Kinesis.

The difficulty will be to detect when all the pieces are available to start the final aggregation. If 50 is a fixed number, that will be easy, otherwise you will need to think about a mechanism to tell the aggregation function it can start to work...