I have an AWS Step Function that consists of a "Map"
(looping over an Array in the input), and then a number of tasks carried out in sequence with all the results. However, in the last step I'd also like to be able to access some of the original input data, that was not used in the intermediate steps. How should I best do this?
What I have tried so far:
Passing all input data through the map tasks. This however fails quickly, because there is a limit on how large results can be passed between functions, and with hundreds of iteration the original input is multiplied hundreds of times.
Using Parallel
state, and add an empty state in one branch of the parallel execution that simply passes the input on, while all of the iteration happens in the other. This feels like a hack, because my step structure now poorly reflects the actual logic of the code.
What would be the “right” way to do this?
Iterate
over Fiz
+--------+ +-------------+ +------------+
{ | +---+ | | | | |
Foo: "Bar", --> -------+ | +---+ Y +----+ Z |
Fiz: [...], | +-+ | | | | |
} +-----+ +-------------+ +------------+
In box Z above, I want to have access to both 'Foo', and the output of Y
It turns out that each step in the SFN has access to the original input, but you have to explicitly state insert them using Parameters
. All the input is available in $$.Execution.Input
.
When using Parameters
, you are overriding all other input, so you will also have to define what you want to pass along from the previous step. In other words: If Y is passing {fruit: "apple"}
to Z, and you want Z to see both that and the original input {Foo: "Bar"}
, you would add the following to the Z step in SFN:
{
"Parameters": {
"Foo.$": "$$.Execution.Input.Foo",
"fruit.$": "$.apple"
}
Note the double $$
in the first case, and the single $
in the second.