AWS Stepfunctions - Copy files between buckets

Question

I need to copy a file from a bucket to another inside a state machine (AWS Step Functions).

I was thinking about create a lambda to do this...

My question is, there is another way to do this without creating a specific lambda to do this? There is a way to run this "s3 cp" command using an specific resource in my Cloud Formation?

Answer 1

You can definitely do this without a Lambda function. You could leverage the S3 integration within Step Functions. Here is a minimal example:

{
  "Comment": "Copy S3 files",
  "StartAt": "Sample input",
  "States": {
    "Sample input": {
      "Type": "Pass",
      "Next": "Map",
      "Result": [
        {
          "Bucket": "target-bucket",
          "CopySource": "source-bucket/source-object-path",
          "Key": "target-object-path"
        }
      ]
    },
    "Map": {
      "Type": "Map",
      "ItemProcessor": {
        "ProcessorConfig": {
          "Mode": "INLINE"
        },
        "StartAt": "CopyObject",
        "States": {
          "CopyObject": {
            "Type": "Task",
            "End": true,
            "Parameters": {
              "Bucket.$": "$.Bucket",
              "CopySource.$": "$.CopySource",
              "Key.$": "$.Key"
            },
            "Resource": "arn:aws:states:::aws-sdk:s3:copyObject"
          }
        }
      },
      "End": true
    }
  }
}

Using map state you can even copy multiple files by passing a list!