AWS Step Functions - Batch Send Data to Child/Nested Step Function State Machine

Question

I am looking for a way to batch send data to a child state machine, from a patent state machine, 50 records at a time.

What I mean by this?

I have two AWS Step Functions state machines. One parent & one child state machine. The parent state machine has the purpose of getting all users from our Jenkins Platform. It looks like this:

Comment: >-
  A state machine that coordinates the end-to-end process for Jenkins
StartAt: Get Jenkins Users
States:
  Get Jenkins Users:
    Type: Task
    Resource: "${JenkinsUsersFunctionArn}"
    TimeoutSeconds: 65
    HeartbeatSeconds: 30
    Retry:
      - ErrorEquals:
          - Lambda.ServiceException
          - Lambda.AWSLambdaException
          - Lambda.SdkClientException
          - States.Timeout
        IntervalSeconds: 2
        MaxAttempts: 3
        BackoffRate: 2
    Catch:
      - ErrorEquals:
          - States.ALL
        ResultPath: "$"
        Next: Process Error
    ResultPath: "$"
    Next: Filter Jenkins Users

This outputs an array of objects, with about 1,500 objects to the $.users object.

What I would like to do is verses sending ALL 1,500 objects down to the child state machine, I would like to only send 50 at a time. Meaning if there was 1,500 objects it would in total invoke 30 state machines.

I have come up with something like this:

  Filter Jenkins Users:
    Type: Map
    InputPath: "$.users"
    MaxConcurrency: 50
    Iterator:
      StartAt: Filter
      States:
        Filter:
          Type: Task
          Resource: arn:aws:states:::states:startExecution
          Parameters:
            StateMachineArn:
            Input: 
              Users: "$"
              AWS_STEP_FUNCTIONS_STARTED_BY_EXECUTION_ID.$: "$$.Execution.Id"
          End: true
    ResultPath: "$.users"
    Next: Success State

But this spins up 50 state machines at the same time, this isn't what I would like. I would like to batch send 50 objects down to the child state machine.

Why Do I want to do this?

I would hit SO many limits if I was to process this all within one state machine, I am trying to split it up to multiple step-function state machines to ensure I don't hit any limits.

Is this possible?

Answer 1

The limit here is based on your whole AWS account per region. So you still also got that limit if you split.

Anyway here is how I would do it.

1. Your lambda instead return 1500 users in one array, let chunk it to 50 users each chunk
2. Use Map state with concurrency of 1 and in put is the chunked array (of 50)

"${JenkinsUsersFunctionArn}" expect to return this

{
  "users": [
    [
      50 users...
    ],
    [
      50 users...
    ],
    ...
    [
      15 users...
    ]
  ]
}

Here is your map state

  Filter Jenkins Users:
    Type: Map
    InputPath: "$.users"
    MaxConcurrency: 1
    Iterator:
      StartAt: Filter
      States:
        Filter:
          Type: Task
          Resource: arn:aws:states:::states:startExecution
          Parameters:
            StateMachineArn:
            Input: 
              Users: "$"
              AWS_STEP_FUNCTIONS_STARTED_BY_EXECUTION_ID.$: "$$.Execution.Id"
          End: true
    ResultPath: "$.users"
    Next: Success State