I have 3 tasks in an airflow dag.
These three tasks have time dependency
Task 1 - 8 am morning
Task - 2 - 10 am morning
Task -3 - 12 am
I'm not finding any doc referring to that. Its telling to set upstream or downstream jobs only. Could anyone helps on this
I'm using Google CLoud Composer
Well, Airflow structure is made so that the schedule_interval
is set at the DAG level. This means that you can set the time when an entire DAG will start its execution, but you cannot really specify different execution times per task.
The solution if you have three separate tasks that are not dependent on each other is to create three different DAGs, and schedule them at those three different times.
If instead the time dependency of task_2
and task_3
is not so important, but you only care that are executed one after the other you can, indeed, set dependencies between the tasks so that task_2
runs always after task_1
has finished and task_3
runs always after task_2
has finished. To set dependencies you can use the really handy syntax (assuming your tasks have been assigned to variables task_1
, task_2
, task_3
):
task_1 >> task_2 >> task_3
You can refer to Airflow official documentation for more information.
TL;DR: You cannot schedule single tasks to be run at different specific times, as the only time you can set is the overall DAG run one.