PROBLEM: We are iterating over list of items and for each item, we are pushing messages into SQS where we have lambda trigger. Suppose, we have a 100 items and we are sending 100 messages into the SQS. The, 100 concurrent lambda functions will be executed( and lambda have concurrency set to 50). Now, We need to send a email when all these 100 lambda functions executed successfully. Is there a way in AWS where we can monitor these 100 lambda function status?
I have few ideas, where we can create a separate row in db for each item and mark status as completed after each lambda successful execution. And, then at the end of every lambda function, we can check if we have all 100 entries in db with status completed, then we can send email.
EDIT: Or, we can have step function where parent task can have dynamic parallel states including all child lambda functions using state machine map state. Then, at the next step, we can send the consolidate email.
Please, let me know your thoughts. That would be very helpful
This looks like a case where Step Functions Distributed Map would help. I'm not sure how you are doing your original iteration, but you could replace that with iteration using Distributed Map. If the list you are iterating over is in S3 already, they you get that out of the box. If it's elsewhere (e.g., in DynamoDB) then you'd need to add a step to either read that list in (if it's less than 256 kB in size, this can be passed in as state input) or write it to S3 first to iterate over (can be in JSON or CSV).
With this, you can then define a workflow to run for each of the items, including calling your Lambda functions. And then when the Distributed Map state completes, you can add in any subsequent steps like sending an email or emitting events to EventBridge.
To learn more about Distributed Map, you can read the docs linked above and check out this blog post or this session from re:Invent 2022.