I am running into the following error when starting up containers on my Raspberry Pi 3B on Raspbian Buster:
docker: Error response from daemon: : signal: illegal instruction (core dumped): unknown.
ERRO[0001] error waiting for container: context canceled
Lately I've been maxing out resources on the Raspberry Pi 3B (looking into upgrading but I don't want to stop selfhosting in the meantime), so sometimes containers are exiting error status due to lack of resources.
Containers have been working fine when restarting until this morning. This morning I've been getting the above illegal instruction error. This happens with all containers I have, docker and docker-compose, and even just when running docker run hello-world using dockers hello-world example container. There hasn't been any updates run in a week, so new updates likely wouldn't be the problem (namely with docker-ce, raspberrypi-kernal which have cause conflicts in the past but were quickly resolved).
I've tried rebooting, removing and installing docker-ce, removing all containers, removing all networks, removing images, and restarting the Docker Service.
When running something like docker-compose --verbose up -d lidarr, the illegal instruction error always seems to occur around connecting to the network. Towards the end of the output is below, with some of the repetition removed:
...
...
compose.cli.verbose_proxy.proxy_callable: docker create_container <- (environment=['PUID=1000', 'PGID=1000', 'TZ=America/New_York'], image='linuxserver/lidarr:latest', ports=[('8686', 'tcp')], volumes={'/music': {}, '/config': {}, '/downloads': {}}, name='pi_lidarr_1', detach=True, labels={'com.docker.compose.project': 'pi', 'com.docker.compose.service': 'lidarr', 'com.docker.compose.oneoff': 'False', 'com.docker.compose.project.working_dir': '/home/pi', 'com.docker.compose.project.config_files': 'docker-compose.yml', 'com.docker.compose.container-number': '1', 'com.docker.compose.version': '1.29.2', 'com.docker.compose.config-hash': '1ee750c8b41eee12fce59c62e06cdf73a46c7177c028d758de86977788a63b1e'}, host_config={'NetworkMode': 'pi_default', 'RestartPolicy': {'Name': 'unless-stopped', 'MaximumRetryCount': 0}, 'VolumesFrom': [], 'Binds': ['/path/to/music:/music:rw', '/path/to/config:/config:rw', '/path/to/downloads:/downloads:rw'], 'PortBindings': {'8686/tcp': [{'HostIp': '', 'HostPort': '8686'}]}, 'Links': [], 'LogConfig': {'Type': '', 'Config': {}}}, networking_config={'EndpointsConfig': {'pi_default': {'Aliases': ['lidarr'], 'IPAMConfig': {}}}})
compose.parallel.feed_queue: Pending: set()
compose.parallel.feed_queue: Pending: set()
...
...
compose.parallel.feed_queue: Pending: set()
compose.parallel.feed_queue: Pending: set()
compose.cli.verbose_proxy.proxy_callable: docker create_container -> {'Id': 'c33e4f4808dac0c794f3806176ad73cafddf10a7088e62181fcf79065db6cb4f',
'Warnings': []}
compose.cli.verbose_proxy.proxy_callable: docker inspect_container <- ('c33e4f4808dac0c794f3806176ad73cafddf10a7088e62181fcf79065db6cb4f')
compose.parallel.feed_queue: Pending: set()
compose.cli.verbose_proxy.proxy_callable: docker inspect_container -> {'AppArmorProfile': '',
'Args': [],
'Config': {'AttachStderr': False,
'AttachStdin': False,
'AttachStdout': False,
'Cmd': None,
'Domainname': '',
'Entrypoint': ['/init'],
'Env': ['PUID=1000',
'PGID=1000',
...
compose.cli.verbose_proxy.proxy_callable: docker disconnect_container_from_network <- ('c33e4f4808dac0c794f3806176ad73cafddf10a7088e62181fcf79065db6cb4f', 'pi_default')
compose.parallel.feed_queue: Pending: set()
compose.cli.verbose_proxy.proxy_callable: docker disconnect_container_from_network -> None
compose.cli.verbose_proxy.proxy_callable: docker connect_container_to_network <- ('c33e4f4808dac0c794f3806176ad73cafddf10a7088e62181fcf79065db6cb4f', 'pi_default', aliases=['lidarr', 'c33e4f4808da'], ipv4_address=None, ipv6_address=None, links=[], link_local_ips=None)
compose.parallel.feed_queue: Pending: set()
compose.parallel.feed_queue: Pending: set()
compose.parallel.feed_queue: Pending: set()
compose.parallel.feed_queue: Pending: set()
compose.cli.verbose_proxy.proxy_callable: docker connect_container_to_network -> None
compose.cli.verbose_proxy.proxy_callable: docker start <- ('c33e4f4808dac0c794f3806176ad73cafddf10a7088e62181fcf79065db6cb4f')
compose.parallel.feed_queue: Pending: set()
compose.parallel.feed_queue: Pending: set()
...
...
compose.parallel.feed_queue: Pending: set()
compose.parallel.feed_queue: Pending: set()
Creating pi_lidarr_1 ... error
compose.parallel.feed_queue: Pending: set()
ERROR: for pi_lidarr_1 Cannot start service lidarr: : signal: illegal instruction (core dumped): unknown
compose.parallel.parallel_execute_iter: Failed: <Service: lidarr>
compose.parallel.feed_queue: Pending: set()
ERROR: for lidarr Cannot start service lidarr: : signal: illegal instruction (core dumped): unknown
ERROR: compose.cli.main.exit_with_metrics: Encountered errors while bringing up the project.
For system info, uname -a outputs:
Linux raspberrypi 5.10.63-v7+ #1496 SMP Wed Dec 1 15:58:11 GMT 2021 armv7l GNU/Linux
docker-compose version outputs:
docker-compose version 1.29.2, build unknown
docker-py version: 5.0.0
CPython version: 3.7.3
OpenSSL version: OpenSSL 1.1.1d 10 Sep 2019
docker --version outputs:
Docker version 20.10.12, build e91ed57
docker info displays:
Client:
Context: default
Debug Mode: false
Plugins:
app: Docker App (Docker Inc., v0.9.1-beta3)
buildx: Docker Buildx (Docker Inc., v0.7.1-docker)
Server:
Containers: 1
Running: 0
Paused: 0
Stopped: 1
Images: 7
Server Version: 20.10.12
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 1
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 7b11cfaabd73bb80907dd23182b9347b4245eb5d
runc version: v1.0.2-0-g52b36a2
init version: de40ad0
Security Options:
seccomp
Profile: default
Kernel Version: 5.10.63-v7+
Operating System: Raspbian GNU/Linux 10 (buster)
OSType: linux
Architecture: armv7l
CPUs: 4
Total Memory: 923.2MiB
Name: raspberrypi
ID: EHO6:EVEP:2WCL:G5HM:S4TO:BOFX:WXTM:GXBA:X5Z6:6YYI:YVMW:QGEZ
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No memory limit support
WARNING: No swap limit support
WARNING: No kernel memory TCP limit support
WARNING: No oom kill disable support
WARNING: No blkio throttle.read_bps_device support
WARNING: No blkio throttle.write_bps_device support
WARNING: No blkio throttle.read_iops_device support
WARNING: No blkio throttle.write_iops_device support
Containers, to name a few, that are "causing" the illegal instruction error:
Update: I checked Journalctl for docker entries after trying to docker run hello-world and found:
Feb 26 13:08:25 raspberrypi dockerd[4602]: time="2022-02-26T13:08:25.253390090-05:00" level=error msg="stream copy error: reading from a closed fifo"
Feb 26 13:08:25 raspberrypi dockerd[4602]: time="2022-02-26T13:08:25.253526860-05:00" level=error msg="stream copy error: reading from a closed fifo"
Feb 26 13:08:26 raspberrypi dockerd[4602]: time="2022-02-26T13:08:26.084248757-05:00" level=error msg="2caf65051f4e40260e125c398541db0969a57ff8544c4ae9010c3e859f43ba25 cleanup: failed to delete container from containerd: no such container"
Feb 26 13:08:26 raspberrypi dockerd[4602]: time="2022-02-26T13:08:26.084429642-05:00" level=error msg="Handler for POST /v1.41/containers/2caf65051f4e40260e125c398541db0969a57ff8544c4ae9010c3e859f43ba25/start returned error: : signal: illegal instruction (core dumped): unknown"
Does anyone know how to diagnose and/or recover from this?
I was able to resolve this, unfortunately I won't be able to find out why this happened.
I tried removing and installing docker-ce and dependencies again. I wasn't able to remove due to containerd.service not stopping. I found it was set to always restart, which would normally make sense. I then ran sudo systemctl disable docker containerd and rebooted. I confirmed those services were no longer running by following journalctl output, looking for the usual restarting and core-dump errors from docker and containerd.
I ran sudo apt remove docker-ce and sudo apt autoremove again, then ran docker's get-docker.sh which reinstalled docker. I then ran sudo systemctl enable docker containerd and sudo systemctl start docker containerd. Docker is the same version it was before and the hello-world container and other containers of mine that wasn't previously running is now running successfully.