Why fails my gitlab-runner (in Docker Container) since a few days, I didn't change anything

Question

I have a Ubuntu 18.04 server. On this server is running an gitlab-ce 12.7.5 instance in a docker container and a gitlab-runner 12.8.0 also in a docker container. Since a few days all the pipelines ar failing with:

*Running with gitlab-runner 12.8.0 (1b659122)
  on ddddd kbuTsA6j
Using Docker executor with image ubuntu:18.04 ...
00:02
Pulling docker image ubuntu:18.04 ...
Using docker image sha256:2c047404e52d7f17bdac4121a13cd844447b74e13063f8cb8f8b314467feed06 for ubuntu:18.04 ...
ERROR: Job failed (system failure): Error response from daemon: OCI runtime create failed: container_linux.go:370: starting container process caused: process_linux.go:459: container init caused: process_linux.go:422: setting cgroup config for procHooks process caused: resulting devices cgroup doesn't match target mode: unknown (executor_docker.go:810:0s)*

The config.toml file of the gitlab-runner container looks like this:

concurrent = 1
check_interval = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "ddddd"
  url = "https://myreplacedname.stratoserver.net:8929/"
  token = "kbuTsA6jgE_GrM"
  tls-ca-file = "/etc/gitlab-runner/certs/myreplacedname.stratoserver.net.crt"
  executor = "docker"
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
  [runners.docker]
    tls_verify = false
    image = "docker:19.03.1"
    privileged = true
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/srv/gitlab-runner/config:/etc/gitlab-runner", "/cache", "/builds:/builds"]
    shm_size = 0

Until a few days ago, everything worked fine. I would be very happy if you could help me. Thank you!

Answer 1

I spent several days trying to find a solution to the same problem. I found out that assigning the value false to the privileged field of your [runners.docker] section solves this problem.

So, in your case, the resulting configuration will look like this:

concurrent = 1
check_interval = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "ddddd"
  url = "https://myreplacedname.stratoserver.net:8929/"
  token = "kbuTsA6jgE_GrM"
  tls-ca-file = "/etc/gitlab-runner/certs/myreplacedname.stratoserver.net.crt"
  executor = "docker"
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
  [runners.docker]
    tls_verify = false
    image = "docker:19.03.1"
    privileged = false
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/srv/gitlab-runner/config:/etc/gitlab-runner", "/cache", "/builds:/builds"]
    shm_size = 0

Of course, to apply these changes, you must restart your gitlab-runner instance after performing these actions. I hope it helps you, too.