Airflow DockerOperator volumes and mounts

Question

We have Airflow running (using Docker compose) with several DAG's active. Last week we updated our Airflow to version 2.1.3. This resulted in an error for a DAG where we use DockerOperator:

airflow.exceptions.AirflowException: Invalid arguments were passed to DockerOperator (task_id: t_docker). Invalid arguments were:
**kwargs: {'volumes':

I found this release note telling me that

The volumes parameter in airflow.providers.docker.operators.docker.DockerOperator and airflow.providers.docker.operators.docker_swarm.DockerSwarmOperator was replaced by the mounts parameter

So I changed our DAG from

t_docker = DockerOperator(
        task_id='t_docker',
        image='customimage:latest',
        container_name='custom_1',
        api_version='auto',
        auto_remove=True,
        volumes=['/home/airflow/scripts:/opt/airflow/scripts','/home/airflow/data:/opt/airflow/data'],
        docker_url='unix://var/run/docker.sock',
        network_mode='bridge',
        dag=dag
    )

to this

t_docker = DockerOperator(
        task_id='t_docker',
        image='customimage:latest',
        container_name='custom_1',
        api_version='auto',
        auto_remove=True,
        mounts=['/home/airflow/scripts:/opt/airflow/scripts','/home/airflow/data:/opt/airflow/data'],
        docker_url='unix://var/run/docker.sock',
        network_mode='bridge',
        dag=dag
    )

But now i get this error:

docker.errors.APIError: 500 Server Error for http+docker://localhost/v1.41/containers/create?name=custom_1: Internal Server Error ("json: cannot unmarshal string into Go struct field HostConfig.HostConfig.Mounts of type mount.Mount")

What am I doing wrong?

Answer 1

The change isn't only in the parameter name it's also a change to Mount syntax.

You should replace

volumes=['/home/airflow/scripts:/opt/airflow/scripts','/home/airflow/data:/opt/airflow/data']

with:

mounts=[
    Mount(source="/home/airflow/scripts", target="/opt/airflow/scripts", type="bind"),
    Mount(source="/home/airflow/data", target="/opt/airflow/data", type="bind"),
]

So your code will be:

from docker.types import Mount
t_docker = DockerOperator(
    task_id='t_docker',
    image='customimage:latest',
    container_name='custom_1',
    api_version='auto',
    auto_remove=True,
    mounts=[
        Mount(source="/home/airflow/scripts", target="/opt/airflow/scripts", type="bind"),
        Mount(source="/home/airflow/data", target="/opt/airflow/data", type="bind"),
    ],
    docker_url='unix://var/run/docker.sock',
    network_mode='bridge',
    dag=dag
)