Search code examples
airflowairflow-scheduler

how to kill airflow scheduler and webserver?

I am new to airflow, tried to run a dag by starting airflow webserver and scheduler. After I closed the scheduler and airflow webserver, the airflow processes are still running.

ps aux | grep airflow shows 2 airflow webserver running, and scheduler running for all dags.

I tried running kill $(ps aux | grep airflow | awk '{print $2}') but it did not help.

I don't have sudo permissions and webserver UI access. enter image description here

Solution

  • If you run Airflow locally and start it with the two commands airflow scheduler and airflow webserver, then those processes will run in the foreground. So, simply hitting Ctrl-C for each of them should terminate them and all their child processes.

    If you don't have those two processes running in the foreground, there is another way. Airflow creates files with process IDs of the scheduler and gunicorn server in its home directory (by default ~/airflow/).

    Running

    kill $(cat ~/airflow/airflow-scheduler.pid)

    should terminate the scheduler.

    Unfortunately, airflow-webserver.pid contains the PID of the gunicorn server and not the initial Airflow command that started it (which is the parent of the gunicorn process). So, we will first have to find the parent PID of the gunicorn process and then kill the parent process.

    Running

    kill $(ps -o ppid= -p $(cat ~/airflow/airflow-webserver.pid))

    should terminate the webserver.

    If simply running kill (i.e., sending SIGTERM) for these processes does not work you can always try sending SIGKILL: kill -9 <pid>. This should definitely kill them.

    NOTE: Starting with Airflow 2.2.0, there is the standalone CLI command that runs all Airflow components, which are needed for local development. The command can be terminated with simple Ctrl-C and it automatically shuts down all running components.