Can I programmatically determine if an Airflow DAG was parsed or triggered?
Case: I need to call a python library outside of any DAG tasks to make some env preparations before tasks run.
Problem: I do need it to be run only when the DAG is triggered (at RUNtime), but not at PARSE time (every 30 seconds).
Thanks!
Googled a solution but nothing similar found...
It looks like there is a new documented solution: https://airflow.apache.org/docs/apache-airflow/stable/howto/dynamic-dag-generation.html - see Optimizing DAG parsing delays during execution section (end of page), New in version 2.4.
Example of the code in DAG body outside any task definition:
#In case “full” parsing is needed (for example in DAG File Processor), dag_id and task_id of the context are set to None.
if get_parsing_context().dag_id is not None:
your runtime-only code to be NOT executed at DAG import time goes here
....