Search code examples
dockerdocker-composerhel

Cannot create container for service xxx: open /var/lib/docker/overlay2/969rf5...fdf-init/merged/etc/resolv.conf: Operation not permitted

while deploying docker images on the production server using docker-compose, I got the following error.

Cannot create container for service xxx: open /var/lib/docker/overlay2/969rf5...fdf-init/merged/etc/resolv.conf: Operation not permitted

What I tried:

  • Changed permissions ( So much so that gave 777 for all the directories involved)
  • Upgraded kernel (Saw somewhere that it could be kernel issue)
  • Removed ACL using setfacl -Rb /var/lib/docker
  • Added the "graph: /var/lib/docker" line in daemon.json
  • Restarted docker service couple of times.
  • Tried running the images individually and not using docker-compose.
  • Tried running with sudo
  • Set the --storage-opt overlay2.override_kernel_check=1 (Since it is rhel and kernel version supported for docker overlay2 is >4.0, so was suggested to override it.)

I failed everytime! Have no idea what the issue is or what it could be. Every time I run the docker-compose file, it creates a new directory in /var/lib/docker/overlay2 with old permissions even though I change the permissions for all the other directories in it. At this point I'm not even sure if it is permissions issue or not. Any help would be appreciated! Thank you!

Solution

  • Finally after trying everything on the internet and nothing working out, the issue was resolved. It was due to the Antivirus that was installed on the server. Had it removed by the client and docker/docker-compose started working absolutely fine.

    To give a little more information on the antivirus, it was some FIM antivirus that was installed on a rhel 3.10-xxx which was blocking docker. It was an older version and thank goodness newer versions are compatible with docker. So that's a wrap, thank you all!