Cannot deploy Docker Swarm stack in rootless mode, mkdir /var/lib/docker: permission denied

I've set up Docker in rootless mode under Ubuntu 20.04 and Debian 11 (in my case, using Ansible and this role). I want to deploy a simple Docker stack to the node via Docker Swarm. No other hosts are involved, just one Swarm node from the same machine, acting as a manager.

I can run this project with Docker and Docker Compose just fine, also in rootless mode. All that changes for the rootless setup is that DOCKER_HOST is overwritten in .bashrc:

export XDG_RUNTIME_DIR="/run/user/1000"
export DOCKER_HOST="unix:///run/user/1000/docker.sock"

When I deploy the stack though, none of the services can start (here is an excerpt of the status):

$ docker stack deploy -c docker-stack.yml demo-stack
$ docker stack ps demo-stack --no-trunc
jig6zyewkem2g225509x91nt5   demo-stack_db.1                  registry.example.com/db:v1.20.2      bullseye   Shutdown        Rejected 15 seconds ago   "mkdir /var/lib/docker: permission denied"   
ox6x5w7du9o5ew2v70g5mfg9e   demo-stack_redis.1               registry.example.com/redis:v1.20.2   bullseye   Shutdown        Rejected 15 seconds ago   "mkdir /var/lib/docker: permission denied"   
ipme447wrrsjc8jw6cpfak4hq   demo-stack_web.1                 registry.example.com/web:v1.20.2     bullseye   Shutdown        Rejected 14 seconds ago   "mkdir /var/lib/docker: permission denied"

The services all error with mkdir /var/lib/docker: permission denied. I suppose that it tries to start them as if the system was using rootful Docker, but it's a rootless installation.

I guess the question is: how do I get the Swarm node (which is the very same machine) to use the correct Docker rootless configuration for launching the services? That would include using the correct DOCKER_HOST configuration.

I am unsure if this is even supposed to work. I hear that overlay networks are not supported, but I am only on one machine, so I don't really need this. I do need Swarm for its usable implementation of secrets (compared to the mock implementation from Docker Compose).

Note that I have the same setup with Docker running in (normal) rootful mode, and there, all services can be started. It's therefore not an issue with the Docker stack file itself.

More details with docker info:

Client:
 Context:    default
 Debug Mode: false

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 12
 Server Version: 20.10.13
 Storage Driver: fuse-overlayfs
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: active
  NodeID: hpzsmez48acse9yo1frnx37fo
  Is Manager: true
  ClusterID: zkv7wsoun193kyvbxe1k3hdph
  Managers: 1
  Nodes: 1
  Default Address Pool: 10.0.0.0/8  
  SubnetSize: 24
  Data Path Port: 4789
  Orchestration:
   Task History Retention Limit: 5
  Raft:
   Snapshot Interval: 10000
   Number of Old Snapshots to Retain: 0
   Heartbeat Tick: 1
   Election Tick: 10
  Dispatcher:
   Heartbeat Period: 5 seconds
  CA Configuration:
   Expiry Duration: 3 months
   Force Rotate: 0
  Autolock Managers: false
  Root Rotation In Progress: false
  Node Address: 127.0.0.1
  Manager Addresses:
   127.0.0.1:2377
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 2a1d4dbdb2a1030dc5b01e96fb110a9d9f150ecc
 runc version: v1.0.3-0-gf46b6ba2
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
  rootless
  cgroupns
 Kernel Version: 5.10.0-13-amd64
 Operating System: Debian GNU/Linux 11 (bullseye)
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 1.936GiB
 Name: bullseye
 ID: 3R5P:2UV6:FIP4:UIJV:TDNQ:35DT:DEDI:SMGN:FDUY:JSWO:FRU6:O2HF
 Docker Root Dir: /home/vagrant/.local/share/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine

WARNING: No cpu cfs quota support
WARNING: No cpu cfs period support
WARNING: No cpu shares support
WARNING: No cpuset support
WARNING: No io.weight support
WARNING: No io.weight (per device) support
WARNING: No io.max (rbps) support
WARNING: No io.max (wbps) support
WARNING: No io.max (riops) support
WARNING: No io.max (wiops) support
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

Solution

The solution is simple: Docker Rootless does not work with Docker Swarm. You can have either, but not both.