Search code examples
airflowdirected-acyclic-graphs

Can I programmatically determine if an Airflow DAG was parsed or triggered?


Can I programmatically determine if an Airflow DAG was parsed or triggered?

Case: I need to call a python library outside of any DAG tasks to make some env preparations before tasks run.

Problem: I do need it to be run only when the DAG is triggered (at RUNtime), but not at PARSE time (every 30 seconds).

Thanks!

Googled a solution but nothing similar found...


Solution

  • It looks like there is a new documented solution: https://airflow.apache.org/docs/apache-airflow/stable/howto/dynamic-dag-generation.html - see Optimizing DAG parsing delays during execution section (end of page), New in version 2.4.

    Example of the code in DAG body outside any task definition:

    #In case “full” parsing is needed (for example in DAG File Processor), dag_id and task_id of the context are set to None. 
    if get_parsing_context().dag_id is not None:   
        your runtime-only code to be NOT executed at DAG import time goes here  
        ....