Search code examples
airflowairflow-scheduler

How to stop Airflow running tasks from 'off' dags

I created some DAGs, ran them and stopped them in the middle of their execution (with the OFF button).

The UI still shows 'Running tasks' for those stopped DAGs though. I tried to set 'clear' to those tasks and not they are in blue, in 'shutdown state'.

I am wondering if those tasks are counted in the total of running tasks, and blocking other tasks from starting (with my current configuration, only 32 tasks can run in parallel). Is there a way to clean completely the DAGs that I don't need anymore and to make sure the tasks are not blocking anything and making Airflow slower?

Thanks!

Solution

  • You can delete all of the dag data from the dag_run and task_instances tables in the meta database.

    You can also do this through the Airflow Webserver UI by navigating to

    Browse -> DAG Runs & Browse -> Task Instances

    And deleting all the records relevant to the dag id.

    One note though is that the task and DAG status fields on the main page may take a while to reflect the changes.