How does retry_exponential_backoff work in Airflow tasks?

Question

How do the parameters retries, retry_delay and retry_exponential_backoff interact? When there is a retry, when is it scheduled? Exponential backoff sounds interesting, but how does it work?

e.g. what happens with these parameters? When are the retries scheduled?

retries=3,
retry_delay=300,
retry_exponential_backoff=True

Answer 1

The code for understanding how Airflow implements exponential backoff can be found in airflow/models/taskinstance.py (link). Exponential backoff will double the wait time between retries after every retry.

Here are some examples of different wait intervals between retries to illustrate how long the scheduler will wait to re-schedule a task for a retry:

retries	retry_delay	retry_exponential_backoff	number of seconds to 1st retry	NoSt 2nd retry	NoSt 3rd retry
3	60	True	60	120	240
3	60	False	60	60	60
1	120	True	120	N/A	N/A
1	120	False	120	N/A	N/A
3	300	True	300	600	1200
3	300	False	300	300	300