Search code examples
airflow-schedulerairflow

How can I check on an SSHOperator with nohup to see if the job is done to complete the dag?

I created a dag that runs a python script on an ec2 instance. The python script takes about 2-3 hours to complete, so I added a nohup command to the SSHOperator. How can I check on interval to see if the ec2 python script job is done? I have thought about updating an environment variable and if the variable doesn't get updated in 3 hours the dag errors out. No sure how do I go about doing it and if there is a specific operator I could use to achieve this task. Alternatively any other best practice advice is dearly appreciated. 

t2 = SSHOperator(
    ssh_conn_id='ec2_worker',
    task_id='execute_script',
    command='nohup python me_script.py &',
    dag=dag)
Solution

  • You could make the script write a success file to some known location when it finishes and then let the SFTPSensor check for it.