We're building an OS image using yocto on Debian which outputs a bzipped volume which we can use as a base image in docker using docker import and we push this image to our registry to use as a base image.
cp build/tmp/deploy/images/raspberrypi4/device.tar.bz2 .
docker import device.tar.bz2 registry/base_image
docker push registry/base_image
We include the base image as part of another docker image:
FROM registry/base_image
ADD target/app.jar app.jar
ADD docker-run.sh run.sh
ENTRYPOINT "./run.sh"
This image is then successfully built by our CI on a linux (Amazon Linux 2) agent, and pushed to the registry. I'm able to pull the image and run it on a Mac with the current version of Docker for Mac.
However, trying to run the same docker image on a linux machine (even on the same linux build agent) results in the following exec format error:
standard_init_linux.go:228: exec user process caused: exec format error
Using an alternative docker image as the base allows the entrypoint to execute, so I'm pretty sure the issue is related to our custom base image.
As docker is largely cross platform, I'm surprised it works on MacOS (intel and m1) but not Linux (tested in Ubuntu and Amazon Linux). I've tried both the Ubuntu and Docker hosted apt repositories for the docker install.
How can I further debug?
The issue here was that the base image was arm based, and that Docker for Mac can run arm images out of the box, even on intel machines.
https://docs.docker.com/desktop/multi-arch/
Docker Desktop provides binfmt_misc multi-architecture support, which means you can run containers for different Linux architectures such as arm, mips, ppc64le, and even s390x.
There's a good write up here for running arm docker images on linux x86 hosts https://matchboxdorry.gitbooks.io/matchboxblog/content/blogs/build_and_run_arm_images.html
After installing QEMU on your host OS, you need to mount the QEMU binary:
docker run -it --name your-container-name -v /usr/bin/qemu-arm-static:/usr/bin/qemu-arm-static your-arm-image