Search code examples
pysparkaws-glue

Is there any way to get the success or failed status of a glue job from inside the pyspark script that is being executed?

Hi I am fairly new to the AWS Glue. Is there any possible way to find the job status of the glue script i am running and update in MySQL. I am trying to get the status of the glue script after it has completed the execution from inside the script.

From the below code i am only getting status as running as updated in the Mysql but it should be success or fail.

myNewJobRun = glue_client.start_job_run(JobName=GLUE_JOB)
status = glue_client.get_job_run(JobName=GLUE_JOB, RunId=myNewJobRun['JobRunId'])
print ("GLUE_JOB run ID: " + myNewJobRun['JobRunId'])
print ("GLUE_JOB run State: " +status['JobRun']['JobRunState'])
Solution

  • You may also want to check the status of the retried job (in case you have retry as enabled). Glue assigns a new JobRunId suffixed with "attempt".