Search code examples
dstairflow

How to properly handle Daylight Savings Time in Apache Airflow?

In airflow, everything is supposed to be UTC (which is not affected by DST).

However, we have workflows that deliver things based on time zones that are affected by DST.

An example scenario:

  • We have a job scheduled with a start date at 8:00 AM Eastern and a schedule interval of 24 hours.
  • Everyday at 8 AM Eastern the scheduler sees that it has been 24 hours since the last run, and runs the job.
  • DST Happens and we lose an hour.
  • Today at 8 AM Eastern the scheduler sees that it has only been 23 hours because the time on the machine is UTC, and doesn't run the job until 9AM Eastern, which is a late delivery

Is there a way to schedule dags so they run at the correct time after a time change?

Solution

  • This question was asked when airflow was on version 1.8.x.

    This functionality is built-in now, as of airflow 1.10.

    https://airflow.apache.org/timezone.html

    Set the timezone in airflow.cfg and dst should be handled correctly.