Search code examples
airflowairflow-schedulerairflow-2.x

How to run selected parallel task at a time and to skip few task


I have a DAG which has 5 parallel task. I don't want to execute them all in one go. I want to pass some flag/value to the task suppose if the flag is set to True it runs and if the flag is set to False it gets skip. Can we do this in Airflow2. The default arguments are very basic one and I have not scheduled the DAG.

My DAG Flow looks something like this

starttask>>5paralleltask>>endtask

And these 5 parallel task I have created using a for loop

Thank you so much in advance


Solution

  • The 5 parallel task that I have are bash operators. To skip few task and to run few task I forced the selected task to fail and to show is as skipped I used exit 99 with my bash command like:

    'echo "running";exit 99'

    There are two steps:

    • First to force the task to fail.
    • Second is to skip the task if it fails

    To force fail the task I have added a list of 5 flags(True/False) if the flag is True the task runs if It is false it get forced failed and then skipped. for skipping the task I used exit 99.

    And it is working as I expected in my case.