Search code examples
dockerdocker-composedocker-volume

docker compose up Failed to mount volumes

I am getting these errors when I run docker-compose up

ERROR: for airflow_webserver_1 Cannot start service webserver: OCI runtime create failed: container_linux.go:346: starting container process caused "process_linux.go:449: container init caused \"rootfs_linux.go:58: mounting \\"/srv/airflow/dags\\" to rootfs \\"/var/lib/docker/overlay2/ccf82b65760c1dc8714db8d6b105ab865183c422e8d7d015988dc157104cce0a/merged\\" at \\"/var/lib/docker/overlay2/ccf82b65760c1dc8714db8d6b105ab865183c422e8d7d015988dc157104cce0a/merged/usr/local/airflow/dags\\" caused \\"not a directory\\"\"": unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type

ERROR: for webserver Cannot start service webserver: OCI runtime create failed: container_linux.go:346: starting container process caused "process_linux.go:449: container init caused \"rootfs_linux.go:58: mounting \\"/srv/airflow/dags\\" to rootfs \\"/var/lib/docker/overlay2/ccf82b65760c1dc8714db8d6b105ab865183c422e8d7d015988dc157104cce0a

I've seen similar posts that say that I should check that the mounted directories should exist and be directories (not files), etc...

I cheched that on my host the path /src/airflow/dags exists and contains several files (the code of my dags from previous container runs).

How can I get rid of this error and why could this happening?. I don't mind to launch containers again from scratch losing the previous volume content.

Here is the docker-compose content:

version: '2.1' services: postgres: image: postgres:9.6 restart: unless-stopped environment: - POSTGRES_USER=airflow - POSTGRES_PASSWORD=airflow - POSTGRES_DB=airflow

webserver:
    build: airflow-server
    restart: unless-stopped
    depends_on:
        - postgres
    environment:
        - LOAD_EX=n
        - EXECUTOR=Local
        - VIRTUAL_HOST=airflow.agatha.com
        - VIRTUAL_PORT=8080
    volumes:
        - /srv/airflow/dags:/usr/local/airflow/dags
        # Uncomment to include custom plugins
        # - /srv/airflow/plugins:/usr/local/airflow/plugins
    ports:
        - "8075:8080"
    command: webserver
    healthcheck:
        test: ["CMD-SHELL", "[ -f /usr/local/airflow/airflow-webserver.pid ]"]
        interval: 30s
        timeout: 30s
        retries: 3
    logging:
        driver: json-file
        options:
            max-size: 50m

networks: default: external: name: nginx-proxy

(the airflow-server folder contais a custom Dockerfile to copy some configurations files)

Here is the Dockerfile

FROM puckel/docker-airflow
USER root

COPY proxy.conf /etc/apt/apt.conf.d/proxy.conf
RUN apt-get update && apt-get install -y gnupg apt-transport-https procps

RUN mkdir /usr/local/airflow/ssh
COPY id_rsa_airflow_user /usr/local/airflow/ssh/id_rsa_airflow_user
RUN mkdir /usr/local/airflow/sparkScripts
COPY FileUnion.py /usr/local/airflow/sparkScripts
COPY ETLDag.py /usr/local/airflow/dags

USER airflow

Full command line output:

> :~/docker/airflow$ sudo docker-compose up WARNING: The Docker Engine
> you're using is running in swarm mode.
> 
> Compose does not use swarm mode to deploy services to multiple nodes
> in a swarm. All containers will be scheduled on the current node.
> 
> To deploy your application across the swarm, use `docker stack
> deploy`.
> 
> Starting airflow_postgres_1 ... done Recreating airflow_webserver_1
> ... error
> 
> ERROR: for airflow_webserver_1  Cannot start service webserver: OCI
> runtime create failed: container_linux.go:348: starting container
> process caused "process_linux.go:402: container init caused
> \"rootfs_linux.go:58: mounting \\\"/srv/airflow/dags\\\" to rootfs
> \\\"/var/lib/docker/aufs/mnt/8ce629cdb64b63552688c4843fff357530fb5a938ac4219ffc0ef753d3b222ac\\\"
> at
> \\\"/var/lib/docker/aufs/mnt/8ce629cdb64b63552688c4843fff357530fb5a938ac4219ffc0ef753d3b222ac/usr/local/airflow/dags\\\"
> caused \\\"not a directory\\\"\"": unknown: Are you trying to mount a
> directory onto a file (or vice-versa)? Check if the specified host
> path exists and is the expected type
> 
> ERROR: for webserver  Cannot start service webserver: OCI runtime
> create failed: container_linux.go:348: starting container process
> caused "process_linux.go:402: container init caused
> \"rootfs_linux.go:58: mounting \\\"/srv/airflow/dags\\\" to rootfs
> \\\"/var/lib/docker/aufs/mnt/8ce629cdb64b63552688c4843fff357530fb5a938ac4219ffc0ef753d3b222ac\\\"
> at
> \\\"/var/lib/docker/aufs/mnt/8ce629cdb64b63552688c4843fff357530fb5a938ac4219ffc0ef753d3b222ac/usr/local/airflow/dags\\\"
> caused \\\"not a directory\\\"\"": unknown: Are you trying to mount a
> directory onto a file (or vice-versa)? Check if the specified host
> path exists and is the expected type ERROR: Encountered errors while
> bringing up the project.

And the airflow-serverdirectory contents (some of the files are not being used)

~/docker/airflow$ ls -l airflow-server/
total 326788
-rw-rw-r-- 1 user user 5008 mar 25 12:16 Dockerfile
-rw-rw-r-- 1 user user 7302 mar 24 15:37 ETLDagCreator.py
-rw-rw-r-- 1 user user 5900 mar 24 16:16 ETLDag.py
-rw-rw-r-- 1 user user 2564 mar 24 15:19 FileUnion.py
-rw-r--r-- 1 user user 334559382 mar 18 11:08 hadoop-3.1.1.tar.gz
drwxrwxr-x 2 user user 4096 mar 19 10:42 hadoop-client-conf
-rwxrwxr-x 1 user user 2026 mar 18 10:40 hbase.service.keytab
-rwxrwxr-x 1 user user 132 mar 18 10:39 hdfs.headless.keytab
-rwxrwxr-x 1 user user 2002 mar 18 10:38 hive.service.keytab
-rw-rw-r-- 1 user user 625 mar 19 16:02 hosts
-rwxrwxr-x 1 user user 1679 mar 20 10:17 id_rsa_airflow_user
-rw-rw-r-- 1 user user 529 mar 18 10:33 krb5.conf
-rw-rw-r-- 1 user user 101 mar 18 10:45 proxy.conf
-rw-rw-r-- 1 user user 58 mar 18 16:47 resolv.conf

Thanks

Solution

  • As the error indicates, you are trying to mount a directory onto a file. That's not a valid bind mount in Linux and so docker fails to create the container. The file is created in your image here:

    COPY ETLDag.py /usr/local/airflow/dags

    If you wanted to copy that into a directory called dags you would need a trailing slash:

    COPY ETLDag.py /usr/local/airflow/dags/

    The directory you are trying to mount on the file comes from your volume mount:

    volumes:
    - /srv/airflow/dags:/usr/local/airflow/dags

    Which as you've shown in your comments is a directory:

    $ ls -ld /srv/airflow/dags
drwxr-xr-x 2 root root 4096 mar 25 12:45 /srv/airflow/dags

    These need to both be files or both be directories. You cannot mix them.