Search code examples
jenkinsairflowairflow-schedulergoogle-cloud-composer

Running tasks in a specific time in airflow


I have 3 tasks in an airflow dag.

These three tasks have time dependency

Task 1 - 8 am morning

Task - 2 - 10 am morning

Task -3 - 12 am

I'm not finding any doc referring to that. Its telling to set upstream or downstream jobs only. Could anyone helps on this

I'm using Google CLoud Composer


Solution

  • Well, Airflow structure is made so that the schedule_interval is set at the DAG level. This means that you can set the time when an entire DAG will start its execution, but you cannot really specify different execution times per task.

    The solution if you have three separate tasks that are not dependent on each other is to create three different DAGs, and schedule them at those three different times.


    If instead the time dependency of task_2 and task_3 is not so important, but you only care that are executed one after the other you can, indeed, set dependencies between the tasks so that task_2 runs always after task_1 has finished and task_3 runs always after task_2 has finished. To set dependencies you can use the really handy syntax (assuming your tasks have been assigned to variables task_1, task_2, task_3):

    task_1 >> task_2 >> task_3
    

    You can refer to Airflow official documentation for more information.


    TL;DR: You cannot schedule single tasks to be run at different specific times, as the only time you can set is the overall DAG run one.