I'm new to AWS step functions. Trying to create a basic ETL flow of glue jobs. Upon completion of state machine definition im able to see the graph being generated , but getting a generic error "There are Amazon States Language errors in your state machine definition. Fix the errors to continue", error message that is not allowing me to proceed.
Here is the code and graph :
{
"Comment": "DRC downstream glue jobs execution step function:slf_aws_can_dbisdel_everyone_drc_amp",
"StartAt": "startFlow",
"States": {
"Comment": "various state types of the Amazon States Language",
"startFlow": {
"Comment": "Pass states are useful when constructing and debugging state machines.",
"Type": "Pass",
"Next": "stg_ods"
},
"stg_ods": {
"Type": "Task",
"Resource": "arn:aws:states:::glue:startJobRun.sync",
"Parameters": {
"JobName": "stage_job_name"
},
"Next": "ods_job"
},
"ods_job": {
"Type": "Task",
"Resource": "arn:aws:states:::glue:startJobRun.sync",
"Parameters": {
"JobName": "main_job_name"
},
"Next": "Wait 3 sec"
},
"Wait 3 sec": {
"Comment": "A Wait state delays the state machine from continuing for a specified time.",
"Type": "Wait",
"Seconds": 3,
"Next": "parallel_stg_adr"
},
"parallel_stg_adr": {
"Comment": "A Parallel state can be used to create parallel branches of execution in your state machine.",
"Type": "Parallel",
"Branches": [
{
"StartAt": "stg_job1",
"States": {
"stg_job1": {
"Type": "Task",
"Resource": "arn:aws:states:::glue:startJobRun.sync",
"Parameters": {
"JobName": "stg_job_name1"
},
"End": true
}
}
},
{
"StartAt": "stg_job2",
"States": {
"stg_job2": {
"Type": "Task",
"Resource": "arn:aws:states:::glue:startJobRun.sync",
"Parameters": {
"JobName": "stg_job_name2"
},
"End": true
}
}
}
],
"Next": "parallel_adr_job"
},
"parallel_adr_job": {
"Comment": "A Parallel state can be used to create parallel branches of execution in your state machine.",
"Type": "Parallel",
"Branches": [
{
"StartAt": "job1",
"States": {
"job1": {
"Type": "Task",
"Resource": "arn:aws:states:::glue:startJobRun.sync",
"Parameters": {
"JobName": "some_glue_job",
"Arguments": {
"--target_table": "some_string_table",
"--calendar_key": "some_string"
}
},
"End": true
}
}
},
{
"StartAt": "job2",
"States": {
"job2": {
"Type": "Task",
"Resource": "arn:aws:states:::glue:startJobRun.sync",
"Parameters": {
"JobName": "some_glue_job",
"Arguments": {
"--target_table": "some_string_table",
"--calendar_key": "some_string"
}
},
"End": true
}
}
}
],
"Next": "end_job"
},
"end_job": {
"Type": "Pass",
"End": true
}
}
}
"Comment": "various state types of the Amazon States Language",
This one at Line 5 seems to be incorrect. "States" map cannot have a "Comment" key. Remove it and then try. Rest of the config looks correct.
Edit 1
If the type of Step Function is Express, ".sync" functions won't work. Try changing the ARN to
"Resource": "arn:aws:states:::glue:startJobRun"
and you should be able to save your Step Function. You will then have to figure out how to setup a different Glue task.