Docker rootless unable to pull images

I am running docker rootless on a server, following https://docs.docker.com/engine/security/rootless/.

With this I am able to pull and run hello-world. But whenever I want to pull a more complex image, I run into the following error:

$ docker pull ubuntu
Using default tag: latest
latest: Pulling from library/ubuntu
83ee3a23efb7: Extracting [==================================================>]  28.57MB/28.57MB
db98fc6f11f0: Download complete 
f611acd52c6c: Download complete 
failed to register layer: ApplyLayer exit status 1 stdout:  stderr: lchown /etc/gshadow: operation not permitted

Output from docker info:

$ docker info
Client:
 Context:    default
 Debug Mode: false

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 1
 Server Version: 20.10.2
 Storage Driver: vfs
 Logging Driver: json-file
 Cgroup Driver: none
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runtime.v1.linux runc io.containerd.runc.v2
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 269548fa27e0089a8b8278fc4fc781d7f65a939b
 runc version: ff819c7e9184c13b7c2607fe6c30ae19403a7aff
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
  rootless
 Kernel Version: 4.15.0-135-generic
 Operating System: Ubuntu 18.04.3 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 16
 Total Memory: 125.9GiB
 Name: ourserver
 ID: ZZ4G:IF5W:HJA4:GBZ4:BOHY:YNLX:EPTA:56OH:REXN:QJ5A:2HL3:KWWX
 Docker Root Dir: /home/honerkam/.local/share/docker
 Debug Mode: false
 Username: dhonerkamp
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine

WARNING: Running in rootless-mode without cgroups. To enable cgroups in rootless-mode, you need to boot the system in cgroup v2 mode.
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

Subids are assigned and newuidmap, newgidmap are installed:

$ grep ^$(whoami): /etc/subuid
honerkam:231072:165537
$ grep ^$(whoami): /etc/subgid
honerkam:231072:165537
$ which newuidmap
/usr/bin/newuidmap
$ which newgidmap
/usr/bin/newgidmap

The only unusual part in this setup I can think of is that I have access to my drive on an mounted filesystem, boot root does not have access to it. I also have root access myself.

Any clues on the cause of this issue would be highly appreciated!

Solution

Copying an answer I received on github if anyone else runs into this issue:

NFS doesn't support CAP_DAC_OVERRIDE, so you need to specify custom {"data-root":"/somewhere-out-of-nfs"} in ~/.config/docker/daemon.json to store the images in a non-NFS location.

See https://www.redhat.com/sysadmin/rootless-podman-nfs for technical details.