The project I'm working on has a docker-compose environment for development, and all of a sudden multiple containers start to have I/O errors like this one:
Error: EIO: i/o error, open '/usr/app/src/components/common/FilterComponent.tsx'
NGINX crashes for the same reason, and so on:
nginx: [emerg] open() "/etc/nginx/nginx.conf" failed (5: Input/output error)
Docker has enough disk space (112GB avail/32GB used), everything else on my Mac seems to work fine. The only way to make it work again is a Docker factory reset, until it happens again (~1-2 days).
This is the docker-compose.yml file, pretty basic if you ask me:
x-env-files: &env-files
env_file:
- docker.env
version: '3.3'
services:
mongo-store:
image: mongo:latest
volumes:
- /data/db
ports:
- 27017:27017
networks:
- backend
command: ['mongod', '--bind_ip', '0.0.0.0']
application-ui:
build:
context: .
dockerfile: Dockerfile-dev
image: application-ui:dev
ports:
- 5000:5000
networks:
- frontend
stdin_open: true
volumes:
- ./src/:/usr/app/src/
command: ['npm', 'run', 'start']
application-service:
build:
context: ../application-service
dockerfile: Dockerfile-local
image: application-service:dev
ports:
- 9000:9000
networks:
- backend
- frontend
environment:
- 'GITLAB_TOKEN=${GITLAB_TOKEN}'
<<: *env-files
volumes:
- ../application-service/src/:/usr/app/src/
nginx:
image: nginx:latest
volumes:
- ./nginx/nginx.local.conf:/etc/nginx/nginx.conf
ports:
- 80:80
- 443:443
networks:
- backend
- frontend
networks:
backend:
frontend:
This is the output of docker run --rm -v /:/host busybox df -h:
Filesystem Size Used Available Use% Mounted on
overlay 149.4G 34.8G 106.9G 25% /
tmpfs 64.0M 0 64.0M 0% /dev
tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
shm 64.0M 0 64.0M 0% /dev/shm
overlay 3.9G 316.0K 3.9G 0% /host
dev 3.8G 0 3.8G 0% /host/dev
shm 3.9G 0 3.9G 0% /host/dev/shm
/dev/vda1 149.4G 34.8G 106.9G 25% /host/etc/cni/net.d
/dev/vda1 149.4G 34.8G 106.9G 25% /host/etc/kubernetes
tmpfs 796.0M 480.0K 795.5M 0% /host/etc/resolv.conf
tmpfs 796.0M 480.0K 795.5M 0% /host/run/config
tmpfs 796.0M 480.0K 795.5M 0% /host/run/desktop
tmpfs 796.0M 480.0K 795.5M 0% /host/run/guest-services
tmpfs 796.0M 480.0K 795.5M 0% /host/run/host-services
cgroup_root 10.0M 0 10.0M 0% /host/sys/fs/cgroup
/dev/vda1 149.4G 34.8G 106.9G 25% /host/usr/libexec/kubernetes/kubelet-plugins
/dev/vda1 149.4G 34.8G 106.9G 25% /host/var/lib
/dev/vda1 149.4G 34.8G 106.9G 25% /host/var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/docker
tmpfs 3.9G 316.0K 3.9G 0% /host/var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/docker/tmp
overlay 3.9G 316.0K 3.9G 0% /host/var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/docker/rootfs
tmpfs 3.9G 4.0K 3.9G 0% /host/var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/acpid/tmp
overlay 3.9G 4.0K 3.9G 0% /host/var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/acpid/rootfs
tmpfs 3.9G 0 3.9G 0% /host/var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/binfmt/tmp
overlay 3.9G 0 3.9G 0% /host/var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/binfmt/rootfs
tmpfs 3.9G 8.0K 3.9G 0% /host/var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/dhcpcd/tmp
overlay 3.9G 8.0K 3.9G 0% /host/var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/dhcpcd/rootfs
tmpfs 3.9G 0 3.9G 0% /host/var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/diagnose/tmp
overlay 3.9G 0 3.9G 0% /host/var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/diagnose/rootfs
tmpfs 3.9G 0 3.9G 0% /host/var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/dns-forwarder/tmp
overlay 3.9G 0 3.9G 0% /host/var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/dns-forwarder/rootfs
tmpfs 3.9G 316.0K 3.9G 0% /host/var/lib/mount-docker-cache/entries/docker.tar/70918e63c378be8683de4fc10c553798322a67941c518075202eab14ca0a8c55/containers/services/docker/tmp
overlay 3.9G 316.0K 3.9G 0% /host/var/lib/mount-docker-cache/entries/docker.tar/70918e63c378be8683de4fc10c553798322a67941c518075202eab14ca0a8c55/containers/services/docker/rootfs
tmpfs 3.9G 0 3.9G 0% /host/var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/http-proxy/tmp
overlay 3.9G 0 3.9G 0% /host/var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/http-proxy/rootfs
tmpfs 3.9G 0 3.9G 0% /host/var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/kmsg/tmp
overlay 3.9G 0 3.9G 0% /host/var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/kmsg/rootfs
tmpfs 3.9G 0 3.9G 0% /host/var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/procd/tmp
overlay 3.9G 0 3.9G 0% /host/var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/procd/rootfs
tmpfs 3.9G 0 3.9G 0% /host/var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/sntpc/tmp
overlay 3.9G 0 3.9G 0% /host/var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/sntpc/rootfs
tmpfs 3.9G 0 3.9G 0% /host/var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/socks/tmp
overlay 3.9G 0 3.9G 0% /host/var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/socks/rootfs
tmpfs 3.9G 0 3.9G 0% /host/var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/trim-after-delete/tmp
overlay 3.9G 0 3.9G 0% /host/var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/trim-after-delete/rootfs
tmpfs 3.9G 0 3.9G 0% /host/var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/volume-contents/tmp
overlay 3.9G 0 3.9G 0% /host/var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/volume-contents/rootfs
tmpfs 3.9G 0 3.9G 0% /host/var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/vpnkit-forwarder/tmp
overlay 3.9G 0 3.9G 0% /host/var/lib/mount-docker-cache/entries/services.tar/de9df97d36780dceb06110a4e27244d44c593b57e18ad39cc450365999e5a4a3/containers/services/vpnkit-forwarder/rootfs
overlay 149.4G 34.8G 106.9G 25% /host/var/lib/docker/overlay2/9f12278ac636d7b67274d5ddb0af91405db4034136b4fa978e6882c884413a41/merged
overlay 149.4G 34.8G 106.9G 25% /host/var/lib/docker/overlay2/9f12278ac636d7b67274d5ddb0af91405db4034136b4fa978e6882c884413a41/merged
tmpfs 64.0M 0 64.0M 0% /host/var/lib/docker/overlay2/9f12278ac636d7b67274d5ddb0af91405db4034136b4fa978e6882c884413a41/merged/dev
shm 64.0M 0 64.0M 0% /host/var/lib/docker/overlay2/9f12278ac636d7b67274d5ddb0af91405db4034136b4fa978e6882c884413a41/merged/dev/shm
tmpfs 3.9G 0 3.9G 0% /host/var/lib/docker/overlay2/9f12278ac636d7b67274d5ddb0af91405db4034136b4fa978e6882c884413a41/merged/sys/fs/cgroup
tmpfs 3.9G 0 3.9G 0% /host/var/log
tmpfs 796.0M 480.0K 795.5M 0% /host/var/run/linuxkit-containerd/containerd.sock
tmpfs 796.0M 480.0K 795.5M 0% /host/var/run/linuxkit-external-logging.sock
grpcfuse 931.5G 14.3G 768.1G 2% /host/host_mnt
/dev/vda1 149.4G 34.8G 106.9G 25% /etc/resolv.conf
/dev/vda1 149.4G 34.8G 106.9G 25% /etc/hostname
/dev/vda1 149.4G 34.8G 106.9G 25% /etc/hosts
tmpfs 3.9G 0 3.9G 0% /proc/acpi
tmpfs 64.0M 0 64.0M 0% /proc/kcore
tmpfs 64.0M 0 64.0M 0% /proc/keys
tmpfs 64.0M 0 64.0M 0% /proc/timer_list
tmpfs 64.0M 0 64.0M 0% /proc/sched_debug
tmpfs 3.9G 0 3.9G 0% /sys/firmware
I've never seen this happen before, it just started a couple of weeks ago. It doesn't happen all the time, everything works fine for hours, sometimes days, and then it crashes. None of my colleagues working on the same project has had the same problem.
Docker Desktop for Mac: v3.6.0 (67351)
Docker Engine: v20.10.8
Compose: v1.29.2
MacOS: v11.5.1 (20G80)
Anyone else experiencing this issue? Any suggestion on how I can further investigate the problem?
It seems that the issue disappeared when I deactivated the option called Use gRPC FUSE for file sharing. I don't know if this is related to a problem on my disk or my machine, but I'm glad it works now.