Is there a way to deploy Airflow with Docker but using different workers?

Question

I have an Airflow deployment using docker-compose as explained on the official Airflow documentation (here).

I am asking myself if it is possible to create a setup that allows different python environments for different tasks. I.e. let's say I have a pipeline for project 1 which has conflicting dependencies with the DAGs of project 2.

Is there a way to just create two different worker images and somehow tell Airflow to which worker container the corresponding tasks should be sent?

Thanks!

Answer 1

I would not go with workflow specific Airflow images, because the approach does not scale. That is, if you had, for example, ten workflows with custom Airflow-worker images each, and you decided to upgrade Airflow, you would have to upgrade each of those custom images.

To deal with conflicting dependencies in Python environments, there are virtual environments. Airflow has PythonVirtualenvOperator that allows executing Python callables inside a new Python virtual environment containing exactly the dependencies the callable needs. This way, in the same workflow (DAG), you can have multiple tasks with completely different set of dependencies each.