Search code examples
visual-studio-codepysparkjupyter-notebookaws-glue

Providing params in AWS Glue interactive session

I am trying to develop an AWS Glue job in pyspark using interactive sessions in AWS Glue. To use job.init, I have seen in examples that the JOB_NAME is passed like so

args = getResolvedOptions(sys.argv, ['JOB_NAME'])
...
job.init(args['JOB_NAME'], args)

a) When I run the notebook in the Glue Studio Editor, I get: GlueArgumentError: the following arguments are required: --JOB_NAME. How do I set the parameter without running the whole job?

b) Ultimately, I want to develop in VS Code. Is there a way to set the JOB_NAME parameter there?

Solution

  • In your Glue Interactive sessions notebook you can add required arguments manually by updating the sys.argv. See the example below:

    sys.argv+=['--JOB_NAME', 'my_test_job']
args = getResolvedOptions(sys.argv,
                          ['JOB_NAME'])
print(args)