DAG is not triggered in requested time

Question

I have this code:

ARGS = {
    'owner': 'airflow',
    'start_date': airflow.utils.dates.days_ago(1),
    'retries': 1,
    'retry_delay': timedelta(minutes=2),
    'retry_exponential_backoff': True,
    'max_retry_delay': timedelta(minutes=10),
    'email_on_retry': False,
    'email_on_failure': True,
    'email': []
}


DAG = DAG(
    dag_id='ietl_tv3_redge_devices',
    default_args=ARGS,
    schedule_interval= '30 7 * * *',
    max_active_runs=1

And for some reason dag is not triggering itself in every day at 07:30 am. Maybe someone have an idea why it's not happening ? Thanks

Answer 1

According to the provided schedule_interval, you want to run the dag daily at 07:30, however the start date is always equals to now - 1 day, which means at day_1 07:30, the start_date will be day_0 07:30 which greater than or equals to execution date (start_interval_date).

When it equals to 07:30 (no pressure on the scheduler, and it tries to schedule the dag exactly at 07:30), the scheduler will create the dag run because for it, start_date <= execution_date.

And when it grater than 07:30 (it's normal, when in most of the cases, the scheduler takes some seconds to schedule the dag run, and it depends on the number of the dags, the frequencies and the number of running schedulers), it will not schedule the run because start_date > execution_date.

Using a variable start_date is a bad practice, and it is not recommended, this parameter is used by the scheduler to decide if the dag run should be run or not when it does scheduling, when you clear the state of a task/run, or when you run a backfill command. You can try providing pendulum.datetime(2023, 7, 1, tz="UTC") as start_date.