Search code examples
docker-composetraefikpodmanrootless

Permission denied trying to use rootless Podman + docker-compose + Traefik with podman.sock

TL:DR: Trying to use rootless Podman with docker-compose through podman socket, and use a Traefik container (talking to podman socket) to proxy traffic to other containers, related to https://stackoverflow.com/a/73774327/1469083

I get permission denied errors, which I can fix with privileged container, which I don't want to use.

Setup

I am running on RHEL 8

$ cat /etc/redhat-release 
Red Hat Enterprise Linux release 8.6 (Ootpa)

Podman came preinstalled, I added docker-compose ("standalone") and podman-docker:

$ curl -SL https://github.com/docker/compose/releases/download/v2.10.2/docker-compose-linux-x86_64 -o /usr/local/bin/docker-compose
$ chmod a+x /usr/local/bin/docker-compose
$ sudo yum install podman-docker

And activated rootless podman socket so that podman and docker-compose can talk to each other:

$ systemctl --user enable podman.socket
$ systemctl --user start podman.socket
$ systemctl --user status podman.socket
$ export DOCKER_HOST=unix:///run/user/$UID/podman/podman.sock
$ echo $DOCKER_HOST
unix:///run/user/1001/podman/podman.sock

I also switched network backend to netavark, DNS did not work without that change

$ podman info |grep -i networkbackend
  networkBackend: netavark

Problems

First I tried the compose stack from https://stackoverflow.com/a/73774327/1469083 with small modifications:

version: "3"
services:
  frontend:
    image: "docker.io/traefik:v2.8"
    ports:
      - "3000:80"
      - "127.0.0.1:3080:8080"
    command:
      - --api.insecure=true
      - --providers.docker
    volumes:
      - /run/user/$UID/podman/podman.sock:/var/run/docker.sock

  backend:
    labels:
      traefik.http.routers.backend.rule: Host(`localhost`)
    image: "tomcat:latest"
    scale: 3

My setup did not appreciate the $UID variable:

WARN[0000] The "UID" variable is not set. Defaulting to a blank string. 
...
Error response from daemon: make cli opts(): error making volume mountpoint for volume /run/user//podman/podman.sock: mkdir /run/user//podman: permission denied

I replaced the volume map with hard-coded UID=1001 (it is the UID of the user running rootless podman, I assumed I should use that one?). Socket looks like this:

ls -la /run/user/1001/podman/podman.sock 
srw-rw----. 1 myrootlessuser myrootlessuser 0 22. 9. 11:28 /run/user/1001/podman/podman.sock


volumes:
  - /run/user/1001/podman/podman.sock:/var/run/docker.sock

But now I get permission denied errors from Traefik trying to connect to /var/run/docker.sock unsuccessfully:

example-docker-compose-frontend-1  | time="2022-09-22T12:04:52Z" level=error msg="Provider connection error Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get \"http://%2Fvar%2Frun%2Fdocker.sock/v1.24/version\": dial unix /var/run/docker.sock: connect: permission denied, retrying in 5.656635591s" providerName=docker

If I change the Traefik-container to privileged: true, this fixes the problem. I don't get the errors anymore, and proxying works like it should.

But, I would prefer not to use privileged containers for security reasons, or at least understand why it is like this.

Questions

  1. How can I make this work with non-privileged Traefik container?

  2. How do I verify that rootless docker/podman socket is working correctly? I've seen commands like this for testing rootful podman socket, but haven't had success on rootless

    $ sudo curl -H "Content-Type: application/json" --unix-socket /var/run/docker.sock http://localhost/_ping

    curl: (7) Couldn't connect to server

  3. Where can I find the documentation for setting up docker socket for rootless Podman? Did I do my setup correctly (systemctl --user enable podman.socket etc.)? I've only been able to find some blogs about this, and advice varies and is often for older Podman versions. For example:

Solution

  • Question 1 and 2

    If you are using

    export DOCKER_HOST=unix:///run/user/$UID/podman/podman.sock

    you are using rootless (unprivileged) Podman (even if you specify privileged: true in the Compose file).

    To use the leaked socket in the container, you need to run podman run with the command-line option --security-opt label=disable.

    Example:

    Start and enable the podman socket

    $ systemctl --user enable --now podman.socket
Created symlink /home/testuser/.config/systemd/user/sockets.target.wants/podman.socket → /usr/lib/systemd/user/podman.socket.

    Test the Docker API service. Result: failure. Curl prints Couldn't connect to server

    $ podman run --rm \
  -v $XDG_RUNTIME_DIR/podman/podman.sock:/var/run/docker.sock \
  docker.io/library/fedora \
    /usr/bin/curl \
      -H "Content-Type: application/json" \
      --unix-socket /var/run/docker.sock \
        http://localhost/_ping
curl: (7) Couldn't connect to server

    Test the Docker API service again, but now add --security-opt label=disable. Result: success

    $ podman run --rm \
  --security-opt label=disable \
  -v $XDG_RUNTIME_DIR/podman/podman.sock:/var/run/docker.sock \
  docker.io/library/fedora \
    /usr/bin/curl \
      -Hs "Content-Type: application/json" \
      --unix-socket /var/run/docker.sock \
        http://localhost/_ping
OK$

    The Docker API service responded with the text string OK.

    (In the command above I also added the curl option -s so that curl prints less debug output)

    I would guess that adding privileged: true to the Compose file has the same effect as providing the --privileged command-line option to podman run.

    One of the effects of using --privileged is that it implies --security-opt label=disable.

    Question 3

    Summary:

    It is enough to run

    $ systemctl --user start podman.socket

    to set up the UNIX socket for rootless Podman.

    It's not necessary but if you in addition to that run

    $ systemctl --user start podman.service

    the podman.service will be started right away (even before the first client has connected to the UNIX socket).

    Longer version:

    If podman.socket is active, then the podman.service will be started when a client connects. (Podman supports socket activation)

    The podman.service will also be started after a reboot if the podman.service has been enabled (systemctl --user enable podman.service) and lingering is enabled (loginctl enable-linger).

    The podman.service will also be started when the user logs in if the podman.service has been enabled (systemctl --user enable podman.service).

    The podman process running in the podman.service will automatically exit after some time of inactivity (by default 5 seconds).

    On a Fedora 36 computer, the Restart directive is set to no (the default value):

    $ grep Restart= /usr/lib/systemd/user/podman.service
$

    This means that it really doesn't matter much whether

    systemctl --user enable podman.service

    has been run or not. The service podman.service will anyway be stopped in 5 seconds if no clients access it.