How to know if the Glue job execution finished successfully when it is triggered from a lambda

Question

I am trying to build a lambda function which execute a Glue job:

import json
import boto3


def lambda_handler(event, context):
    
    client = boto3.client('glue')
    
    response = client.start_job_run(
               JobName = 'job1',
               Arguments = {
                 '--parametro1': 'par 1'} )
    
    
    return {
        'statusCode': 200,
        'body': json.dumps('Hello from Lambda!')
    }

The lambda1 does not wait for the job execution finishes, it only execute the job and finishes.

I would like the lambda handles the result of job execution, because the lambda is executed from a step function, which only if the lambda finishes successfully, the step function goes to the next step.

Answer 1

I think you should just be using the arn:aws:states:::glue:startJobRun.sync service integration from Step Functions. This will run your glue job, monitor for completion, and then allow you workflow to continue when it’s done. You don’t need a Lambda function in the middle nor to implement the polling for completion.

You can learn more here: https://docs.aws.amazon.com/step-functions/latest/dg/connect-glue.html