I have a list of items. The length of this list is unknown. I would like to use a loop to perform some fairly heavy processing on each item. I am wondering if it is possible to do this using a state machine. More specifically, can I have one step of the machine perform the loop and start an instance of the next step per item? If not what would you suggest? The goal is to keep this as serverless as possible.
P.S. I know that I can iterate with step functions, but the state machine will time out after a few minutes, and I am afraid that wouldn't be enough time to process all the items, if the processing is done serially.
By "serverless", i hope u meant not starting & terminating instances manually everyday (or periodically).
Option 1: Put the items in AWS SQS, launch an instance, which will process the item one by one and terminate after all items are done. Here adding the items to SQS and launching the instance can be automated, depending on the trigger point of how/where the item list is coming from.
Option 2: AWS Batch, which can process all items in parallel if that works for your use-case.
Additional components would be needed depending on the trigger point e.g. if you want to fetch the item list from a URL every 6 hours (like a feed), then add a cloudwatch scheduled event, which triggers a lambda function, which downloads the items and uses option 1 or 2.