Search code examples
postgresqlairflowairflow-schedulerpostgresql-12

Airflow scheduler fails to start tasks

My problem:

Airflow scheduler is not assigning tasks.

Background:

I have Airflow running successfully on my local machine with sqlitedb. The sample dags as well as my custom DAGs ran without any issues. When I try to migrate from sqlite database to Postgres (using this guide), the scheduler no longer seems to be assigning tasks. The DAG get stuck on "running" state but no task in any DAGs ever gets assigned a state.

Troubleshooting steps I've taken

  • The web server and the scheduler are running
  • The DAG is set to "ON".
  • After running airflow initdb, the public schema is populated with all of the airflow tables.
  • The user in my connection string owns the database as well as every table in the public schema.

Scheduler Log

The scheduler log keeps posting out this WARNING but I have not been able to use it to find any useful information aside form this other post with no responses. 

[2020-04-08 09:39:17,907] {dag_processing.py:556} INFO - Launched DagFileProcessorManager with pid: 44144
[2020-04-08 09:39:17,916] {settings.py:54} INFO - Configured default timezone <Timezone [UTC]>
[2020-04-08 09:39:17,927] {settings.py:253} INFO - settings.configure_orm(): Using pool settings. pool_size=5, max_overflow=10, pool_recycle=1800, pid=44144
[2020-04-08 09:39:19,914] {dag_processing.py:663} WARNING - DagFileProcessorManager (PID=44144) exited with exit code -11 - re-launching

Environment

  • PostgreSQL version 12.1
  • Airflow v1.10.9
  • This is all running on a MacBook Pro (Catalina) in a conda virtual environment.
Solution

  • Postgres was installed using postgresapp. Updated postgresapp to version 2.3.3e. PostgresSQL is still version 12.1 but by updating the postgresapp, the issue was solved.