Can I fetch execution logs programmatically from AWS Step Functions?

Question

My team has a Step function deployed in multiple geo distributed AWS accounts. We want to aggregate the execution logs from each of these step function instances into 1 data store. I was thinking of using a Lambda that would call the GetExecutionHistory API https://docs.aws.amazon.com/step-functions/latest/apireference/API_GetExecutionHistory.html, but the problem here becomes handling dedup - to avoid this we would have to only call the API once we know the step function is done, but this will be slow and that means there will be a big backlog of executions to process.

Rather, it would be great if there is a way to get notified of all execution events, INCLUDING start of execution and end of execution. That way, all events - hopefully associated with the event id itself, can be sent some queue. Then, I can have a reader lambda read from this and add to my backend store.

Any ideas/options around this?

Answer 1

You can create a Event bridge with.

{ "source": ["aws.states"] }

You can have lambda as target of this event that will parse step function and save to S3 or where ever you want This will work for every step changes change .