I have a DAG with multiple tasks. When the DAG is executed by scheduler it enters running
state but none of tasks in the DAG has any status. However, if the same DAG is executed manually (Trigger Dag
button click) - tasks become queued->running->success
.
I have tried to check Stackdriver logs for "Cloud Composer Environment" - but haven't found any suspicious log entries. Also have checked task logs in airflow UI - they were empty.
Why are scheduled dag runs stuck? How can I figure out reasons of scheduled DAGs problems?
Reason was start_date
param. It was calculated with this way:
START_DATE = datetime.datetime.now() - datetime.timedelta(hours=1)
dag = airflow.DAG(
'data_transfer_dag',
'catchup=False',
default_args = {
'owner': 'Zufar',
'depends_on_past': False,
'email': [''],
'email_on_failure': False,
'email_on_retry': False,
'retries': 1,
'retry_delay': datetime.timedelta(minutes=5),
'start_date': START_DATE,
},
schedule_interval=datetime.timedelta(hours=1)
)
Fix was to use hardcoded day: START_DATE = datetime.datetime(2020,1,1)
. So it can't be calculated based on current time. Just use hardcoded past date.