I'm working on a step function on AWS and the function will iterate through several items. For each iteration I need to have a lambda function execute some code and output a custom object. I am thinking of having an array at the top level of the step function so after each iteration the objects can be appended to this array and at the end of the execution have this array of all the objects.
Is there a way to have this type of variable? I've read that there are objects (JSON format) that can be passed down from step to step but this will be a quite big array
State Machine executions typically use JSON to communicate state input and output between tasks.
If the items in the input array can be processed independently from one another, the idiomatic solution would be an (inline) Map State. A Map state processes each arbitrary item of an input array, say ["a", "b", "c"]. Processing is concurrent by default. The state's output is an array of arbitrary results of the processor function for each item, e.g.: [{"result": "a-result"}, {"result": "b-result"}, {"result": "c-result"}].
There is a hard limit of 256 KB on the size of the UTF-8 string input/output from a task, state or execution. A second type of Map state, distributed mode, let's you work around this size limit by outputting results to S3.