Search code examples
aws-glue

"Command should not be null" error updating default argument in AWS Glue job

I'm trying to update programatically the parameter that I've introduced in an AWS Glue job in the following way:

aws glue update-job --job-name myjobname --job-update '{"DefaultArguments" : {"MONTH" : "202106"} }'

But gives me this error:

An error occurred (InvalidInputException) when calling the UpdateJob operation: Command should not be null

I've also tried with double dash in the key property, but gives the same error.

Any idea on how to solve this?

Thanks in advance!

Solution

  • Per the guidance here: https://awscli.amazonaws.com/v2/documentation/api/latest/reference/glue/update-job.html
    The "Command" argument is required (see above link for more details)
    I believe that the role argument is also required.

    Assuming you are running a typical python3 Apache spark job, you command should look something like:

    aws glue update-job --job-name myjobname --job-update '{"Role":"[your role]", "Command":{"Name": "glueetl ", "ScriptLocation": "[path to your script]", "PythonVersion": "3"},"DefaultArguments" : {"MONTH" : "202106"} }'

    I would be cautious with the update-job command, I believe that it replaces all unspecified arguments with blanks, meaning you will need to include everything you don't want to lose within the command. Please be sure to validate before use in a production environment.