Search code examples
celeryairflow

Adding extra celery configs to Airflow

Anyone know where I can add extra celery configs to airflow celery executor? For instance I want http://docs.celeryproject.org/en/latest/userguide/configuration.html#worker-pool-restarts this property but how do I allow extra celery properties..

Solution

  • Use the just-released Airflow 1.9.0 and this is now configurable.

    In airflow.cfg there is this line:

    # Import path for celery configuration options
celery_config_options = airflow.config_templates.default_celery.DEFAULT_CELERY_CONFIG

    which points to a python file from the import path. The current default version can is https://github.com/apache/incubator-airflow/blob/1.9.0/airflow/config_templates/default_celery.py

    If you need a setting that isn't tweakable via that file then create a new module, say 'my_celery_config.py':

    CELERY_CONFIG = {
    # ....
}

    and put it in your AIRFLOW_HOME dir (i.e. along side the dags/ folder) and then set celery_config_options = my_celery_config.CELERY_CONFIG in the config.