I want to create AWS GLUE job(GLUE version 4.0) with cloudformation, but have this error:
Glue Version 4.0 is only supported for Glue ETL, Ray and Streaming jobs.
Here is my template resource code
TestJob:
Type: AWS::Glue::Job
Properties:
Connections:
Connections:
- !Sub job-${Env}
Command:
Name: !Sub job-${Env}
ScriptLocation: !Ref JobScriptLocation
ExecutionProperty:
MaxConcurrentRuns: 1
MaxRetries: 3
DefaultArguments:
"--enable-metrics": true
"--enable-spark-ui": true
"--extra-py-files": s3://aws-glue-studio-transforms-560373232017-prod-eu-central-1/gs_common.py,s3://aws-glue-studio-transforms-560373232017-prod-eu-central-1/gs_derived.py
"--spark-event-logs-path": s3://test/sparkHistoryLogs/
"--enable-job-insights": false
"--enable-glue-datacatalog": true
"--enable-continuous-cloudwatch-log": true
"--job-bookmark-option": job-bookmark-disable
"--job-language": python
"--TempDir": s3://test/temporary/
Name: !Sub job-${Env}
Role: !Ref Role
WorkerType: "G.1X"
NumberOfWorkers: 5
GlueVersion: 4.0
The property Name: !Sub job-${Env}
for Command
is incorrect.
Name - The name of the job command. For an Apache Spark ETL job, this must be glueetl. For a Python shell job, it must be pythonshell. For an Apache Spark streaming ETL job, this must be gluestreaming. For a Ray job, this must be glueray.