Search code examples
dockerairflowdocker-volumedockeroperator

Airflow DockerOperator, how to add a volume to a container

I have a serie of tasks that is supposed to spin up containers and run a python script in them. I however need to mount the volume which contains the python code...

After running into many errors I found out adding a volume is not supported anymore, and now a Mount object has to be specified: https://github.com/apache/airflow/pull/15843

My code looks like this for the task:

from airflow.operators.docker_operator import DockerOperator
from docker.types import Mount

code_dir = Mount(target='/SRC',
                     source='/SRC/code',
                     type='bind')

task_name = DockerOperator(
            task_id=f"task_{task_name}",
            image='python-multi-purpose:latest',
            container_name=task_name,
            mount=[code_dir],
            api_version='auto',
            auto_remove=True,
            command=command,
            docker_url="unix://var/run/docker.sock",
            network_mode="bridge",
            dag=dag
        )

Unfortunately I am running in the same exact error I was receiving when using volumes=[list]:

airflow.exceptions.AirflowException: Invalid arguments were passed to DockerOperator (task_id: task_yh_get_info.A). Invalid arguments were:
**kwargs: {'mount': [{'Target': '/SRC', 'Source': '/SRC/code', 'Type': 'bind', 'ReadOnly': False}]}

Would anyone be able to provide the syntax and a logical explanation of how to make this work?

Alternatively, any suggestion on how to approach this? keeping in mind I have Airflow running inside a container

Thanks!

Solution

  • OK. Another answer then (if you have the latest docker provider):

    mount -> mounts.

    You have typo in parameter name: https://airflow.apache.org/docs/apache-airflow-providers-docker/stable/_api/airflow/providers/docker/operators/docker/index.html