Search code examples
amazon-web-servicesaws-cloudformationaws-glue

I can't create AWS glue job 4.0 with cloudformation


I want to create AWS GLUE job(GLUE version 4.0) with cloudformation, but have this error:

Glue Version 4.0 is only supported for Glue ETL, Ray and Streaming jobs.

Here is my template resource code

  TestJob:
    Type: AWS::Glue::Job
    Properties:
      Connections:
        Connections:
          - !Sub job-${Env}
      Command:
        Name: !Sub job-${Env}
        ScriptLocation: !Ref JobScriptLocation
      ExecutionProperty:
        MaxConcurrentRuns: 1
      MaxRetries: 3
      DefaultArguments:
        "--enable-metrics": true
        "--enable-spark-ui": true
        "--extra-py-files": s3://aws-glue-studio-transforms-560373232017-prod-eu-central-1/gs_common.py,s3://aws-glue-studio-transforms-560373232017-prod-eu-central-1/gs_derived.py
        "--spark-event-logs-path": s3://test/sparkHistoryLogs/
        "--enable-job-insights": false
        "--enable-glue-datacatalog": true
        "--enable-continuous-cloudwatch-log": true
        "--job-bookmark-option": job-bookmark-disable
        "--job-language": python
        "--TempDir": s3://test/temporary/
      Name: !Sub job-${Env}
      Role: !Ref Role
      WorkerType: "G.1X"
      NumberOfWorkers: 5
      GlueVersion: 4.0

Solution

  • The property Name: !Sub job-${Env} for Command is incorrect.
    Name - The name of the job command. For an Apache Spark ETL job, this must be glueetl. For a Python shell job, it must be pythonshell. For an Apache Spark streaming ETL job, this must be gluestreaming. For a Ray job, this must be glueray.