Search code examples
pythonairflowjinja2

Airflow: how to use dag_run argument value as trigger_rule in PythonOperator

I am trying to pass the trigger_rule of a PythonOperator via the config when triggering my DAG. For example: {"trigger_rule": "all_done"}. In other words, I want to be able to at runtime choose which trigger_rule to use by using the config when triggering the DAG.

However, it seems that the trigger_rule field in the PythonOperator is not a templated field. I.e. I can not implement it as follows:

PythonOperator(
   task_id='test',
   python_callable=my_func,
   trigger_rule="{{ dag_run.conf.get('trigger_rule', 'all_success') }}"
)

Any clues on how to tackle this problem?

Solution

  • This is not possible as trigger_rule is a parameter needed by the Scheduler to determine if a task should run; meaning it needs to be present prior to task runtime. Template fields are evaluated/rendered during task execution.

    In essence, if a parameter is needed for the Scheduler it cannot be templated.