What is a good way to wait for workers when using AWS Lambda for parallel processing?

Question

Problem

I have an optimization problem and I'm using several instances of AWS Lambda to run the optimization algorithm in parallel, each instance having different parameters. I need to wait for all those workers to finish running, then go through all the results in order to pick the best one.

Note: I'm using a completely serverless architecture.

First idea

My first idea was to use another Lambda function to launch all the workers, wait for them, then collect the results. However, I find this synchronous approach rather inconvenient, because it requires the main function to be running during the whole process, without actually doing any useful work. Ideally I would have one function to launch the workers and a second one to collect the results, which would be triggered only after all the workers were finished.

Question

What is a good way to wait for all the Lambda workers to finish, before triggering the code that works on their results?

Answer 1

You can use Step Functions:

Step Functions is a serverless orchestration service that lets you combine AWS Lambda functions and other AWS services to build business-critical applications.

Specifically, Step Functions supports a Parallel state:

A Parallel state causes AWS Step Functions to execute each branch, starting with the state named in that branch's StartAt field, as concurrently as possible, and wait until all branches terminate (reach a terminal state) before processing the Parallel state's Next field.

Alternatively, consider the Map state:

The Map state can be used to run a set of steps for each element of an input array. While the Parallel state executes multiple branches of steps using the same input, a Map state will execute the same steps for multiple entries of an array in the state input.

So, Map supports dynamic parallelism while Parallel supports pre-defined static parallelism.