Search code examples
airflowairflow-taskflow

Can I use airflow tags to determine if all DAGs with a certain tag has finished running?

I have a number of individual DAGs that are not connected to each other. The one thing that 'connects' them are that they are tagged with US_TAG.

I am trying to run a new task after all of these DAGs (with the US_TAG) have finished running. I was wondering if it is possible to use the US_TAG to do/know?

Any information is much appreciated.

Solution

  • In my understanding, you couldn't use tag to describe dependency between DAGs. Tag is only used for management of tasks.

    If you wish to run DAG1 based on DAG2, you should use ExternalTaskSensor

    Define previous task

    # Task to be depended
task_to_depend = BashOperator(
    task_id='task_to_depend',
    bash_command='echo "This is the task to depend on."',
    dag=dag,
)

    Define new task which depends on it

    from airflow.sensors.external_task_sensor import ExternalTaskSensor

wait_for_task = ExternalTaskSensor(
    task_id='wait_for_task',
    external_dag_id='dag_to_depend_on',  # depended DAG ID
    external_task_id='task_to_depend',  # depended task ID
    dag=dag,
)

wait_for_task >> some_other_task