Airflow retry of multiple tasks as a group

Question

I have a group of tasks that should run as a unit, in the sense if any of the tasks from the group fail, the whole group should be marked as failed.

I want to be able to retry the group when it has failed.

For example, I have a DAG with these tasks:

taskA >> (taskB >> taskC) >> taskD

I want to say that (taskB >> taskC) is a group.

If either taskB or taskC fails, I want to be able to rerun the whole group (taskB >> taskC).

Answer 1

This is a two parts question.

First, In Airflow downstream task can not effect upstream task. Assuming structure of:

taskA >> taskB >> taskC >> taskD

then if taskB is successful and taskC failed. it can not change the state of taskB to failed.

For Airflow>=2.5.0:

There is an option to clean a TaskGroup (see PR)

For Airflow<2.5.0:

Clearing (rerun) a TaskGroup is a feature that is not available.