Search code examples
apache-sparkhadoop-yarnairflow

How can I stop spark-submit jobs on yarn in airflow with SSHOperator

I use airflow version 2.1.0 and here is my code:

task1 = SSHOperator(
ssh_hook=sshHook,
task_id='test_spark,
do_xcom_push=False,
command=command spark submit)

I submit on mode local and kill it -> work ok. But I run on mode yarn it when i kill it on UI airflow it doesn't work. can anyone help me? Thanks for reading.

Solution

  • If you run it on yarn you need to execute the yarn command with the application id of the application.

    The yarn command itself is :

    yarn application -kill enter_application_id

    So you should try something like :

    task1 = SSHOperator(
    ssh_hook=sshHook,
    task_id='test_spark,
    do_xcom_push=False,
    command=yarn application -kill enter_application_id)

    For more information, yarn is a ressource negociator and airlfow is an orchestration tool. When you click on UI you kill the task on the orchestration tool (airflow), but yarn that instanciated the node for your job to run isn't aware of that. Therefor you need to notice yarn that you want to kill the job.