I am thinking of using this solution to process about 3000 objects from S3.
https://docs.aws.amazon.com/step-functions/latest/dg/sample-map-state.html
My use case is that I need to loop through all items one at a time, passing each file to multiple lambda functions for processing, until all items have been processed. I haven't been able to find any information to tell me if there are limitations to this approach. All the examples I have seen use a small number of items and I was wondering if there was any reason that I shouldn't use it for 3000 (or more) items.
AWS Step Functions has 25,000 events limit for maximum execution history size. So if you have 5 steps for processing one file you can't process more than 5000 items.
To overcome this limit:
Wrap all the process for the file in one step functions and call if from your parent step functions (Nested Step Functions).
Instead of passing 1 file to your map or nested step functions, pass a batch of them and them iterate over them in your child state machine. So for example if your batch 10 files, instead of 5000 you can process 50000.
The other workaround is to start another Execution when you reached the 25000 limits, but solving it with the first 2 options is easier.