Is there any way to read the contents of an S3 file from an AWS Step Function?

Question

I have a particular workflow where I want to pass a list of 500 json strings from a lambda function to a step function (stepFunction1), and then iterate over the list in that step function's map state. From there, I want to pass each item in the list to a separate step function (stepFunction2) where additional work will be done.

My problem is that my list of 500 json strings exceeds the AWS service limit when passed to stepFunction1. I have tried splitting up the list into several smaller segments, but this leads to several invocations of stepFunction1 running concurrently, which I can't have due to other limitations. My next idea was to try and store the list of json strings on an S3 bucket, access it from stepFunction1, and then iterate through it from there. Is there any way to achieve this? Is it possible to read a file in S3 from an AWS state machine? I'm a bit stumped here.

Answer 1

One solution is to store the items in an Amazon DynamoDB table and directly access them from AWS Step Functions.

Here's an example how to retrieve an item from DynamoDB:

"Read Next Message from DynamoDB": {
  "Type": "Task",
  "Resource": "arn:aws:states:::dynamodb:getItem",
  "Parameters": {
    "TableName": "MyTable",
    "Key": {
      "MessageId": {"S.$": "$.List[0]"}
    }
  },
  "ResultPath": "$.DynamoDB",
  "Next": "Do something"
}

You can find more information about calling DynamoDB APIs with Step Functions in the documentation.