can we parameterize the airflow schedule_interval dynamically reading from the airflow variables instead of passing directly as the cron expression
I have passed the following way as per airflow documentation args = { 'owner': 'pavan', 'depends_on_past': False, 'start_date': datetime(2020, 1, 15), 'email_on_failure': True, 'email_on_retry': False, 'retries': 0, 'on_failure_callback': notify_email }
with DAG(dag_id=DAG_NAME, default_args=args, schedule_interval='* 1 * * *', catchup=False) as dag:
Yes
Technically you can do it, but it brings 2 problems
minor problem: reading a Variable means a SQL-query being fired to Airflow's SQLAlchemy
backend meta-db. And doing it in your DAG-definition script means this will happen as the DAG is continuously parsed by Airflow in background. Read point 2 here
major problem: A Variable can be edited via UI. But altering an Airflow
DAG's schedule_interval
can have wierd behaviours and may require you to either rename the DAG or (anecdotal finding) restart the scheduler
to fix that