Search code examples
airflowairflow-scheduler

Airflow wait dependency instead of skipping dependency

is there a way to create just wait dependency between tasks, because, I do have a task array for example:

tasks_name = [task1, task2, task3]

They are calling like that:

    task1 >> task2
    task2 >> task3
    task2 >> task_4
    for task3 in tasks_name:
        [task3] >> task4
    task4 >> task5
    task5 >> task6

but if I remove dependency between task4 and task3, it is not running after all tasks_name finish: enter image description here

and if I add this dependecy, it will skip, and I do not want to skip, if I run at least one task3 enter image description here

I am using BranchOperator to validate if it skips or not.

I need a help,

1- If I can run task 4 if I have at least one task3 running

2- If I can add dependecy between task 3 and task 4 to run just when all task 3 have finished

Solution

  • I've found the solution!!!

    I am using the first option

    1- If I can run task 4 if I have at least one task3 running

    Using the param trigger_rule="none_failed" because the default is trigger_rule="all_success" so when it found a skip status, it skip the rest of the flow.

    The code working as expected is like:

    conditional_validate = BranchPythonOperator(
        task_id='conditional_validate',
        python_callable=conditional_validate,
        provide_context=True,
        trigger_rule="none_failed",
        dag=dag)

    enter image description here