Search code examples
aws-glueaws-step-functions

How to read the input state in a Step Function from a Glue Python job?


I have a step function that generates the following input for the next step:

{
  "name": "Done",
  "input": [
    {
      "success": "123"
    },
    {
      "failing": "123"
    }
  ],
  "inputDetails": {
    "truncated": false
  }
}

where the fields in "input":[...] are the output of other steps.

The step function for my job is

"Done": {
      "Type": "Task",
      "Resource": "arn:aws:states:::glue:startJobRun.sync",
      "Parameters": {
        "JobName": "done",
        "Arguments": {
          "--job-output-states": "$$.input"
        }
      },
      "End": true,
      "Credentials": {
        "RoleArn": "arn:aws:iam::1234567890:role/glue_test"
      }
    }

and the job code is

import sys
import boto3
from awsglue.utils import getResolvedOptions

args = getResolvedOptions(sys.argv, [
    "job-output-states"
])

print(args)

The arguments passed to the Glue Python Job seem not parsed. How can I access from there to "input":[...]?


Solution

  • This is the solution that worked for me

    "Done": {
          "Type": "Task",
          "Resource": "arn:aws:states:::glue:startJobRun.sync",
          "Parameters": {
            "JobName": "done",
            "Arguments": {
              "--job-output-states.$": "States.JsonToString($)"
            }
          },
          "End": true,
          "Credentials": {
            "RoleArn": "arn:aws:iam::1234567890:role/glue_test"
          }
        }