Multiplatform Docker Images on Multiple Hosts

I have in front of me two computers, one Linux/x86_64, one MacOS/arm64 (M2 Mac). I want to build a multiplatform Docker image. I'd like to start the build process on one computer for its own architecture (say x86_64), upload/push the built image, then possibly pull that image on the other computer, build the for that architecture (arm64), and push the composite image with multiple architectures in it. I'm interpreting this line from the Docker manual:

When you run an image with multi-platform support, Docker automatically selects the image that matches your OS and architecture.

to mean that there is indeed one multi-platform image entity, not two images in two repositories loosely linked by a manifest.

I don't want to build using QEMU. I don't want to buy into the whole cloud-builder thing. What I want to do seems simple, but I can't seem to find any information on how to do this. All information is about using QEMU or native builders in the cloud, orchestrated in some fashion. Is there a way to follow the simple approach above?

Solution

to mean that there is indeed one multi-platform image entity, not two images in two repositories loosely linked by a manifest.

This is confusing or perhaps inaccurate. A multi-platform image in a registry is represented by a manifest list or index, which is a json structure that contains a list of manifests (as the name implies). Those manifests are all referenced by digest and are located in the same repository. The end result looks like:

You can create and push a single platform image to a registry with the standard docker build. To build a multi-platform image, the easiest option for me is leveraging buildkit and buildx, which by default leverages qemu when running:

docker buildx build --platform=linux/amd64,linux/arm64 -t $repo:$tag .

Buildx even offers the ability to create multiple nodes and build each platform on the appropriate node using docker buildx create --append:

docker context create arm64-build --docker "host=ssh://user@my-arm64-box"
docker buildx create --driver=docker-container multi-platform
docker buildx create --append arm64-build multi-platform
docker buildx use multi-platform

You can skip qemu and leverage cross compiling if the compiler supports it with a few other options in the Dockerfile:

# syntax=docker/dockerfile:1

# --platform=$BUILDPLATFORM runs this step on the local architecture
FROM --platform=$BUILDPLATFORM ${base_image} as build
COPY . /src
WORKDIR /src
# builtin vars are exposed to the environment for the target to build
ARG TARGETOS
ARG TARGETARCH
# the make command would need to leverage the TARGETOS and TARGETARCH vars
RUN make app

# this step runs with the target architecture
FROM ${deploy_image} as release
# with only a copy step, and no run commands, there is no qemu involved
COPY --from=build /src/bin/app /usr/local/bin/app

The third option is to build each image separately, push to the registry, and then create a manifest list that references those two images. There are a few tools for that, but I wouldn't necessarily call this the easy option since so few do it (you are also dealing with race conditions in build pipelines).

Docker's built-in tool for this is docker manifest which has options to create and push the created manifest. It is marked as experimental, so depending on it could mean needing to modify your scripts in future docker versions:

$ docker manifest --help

Usage:  docker manifest COMMAND

The **docker manifest** command has subcommands for managing image manifests and
manifest lists. A manifest list allows you to use one name to refer to the same image
built for multiple architectures.

To see help for a subcommand, use:

    docker manifest CMD --help

For full details on using docker manifest lists, see the registry v2 specification.

EXPERIMENTAL:
  docker manifest is an experimental feature.
  Experimental features provide early access to product functionality. These
  features may change between releases without warning, or can be removed from a
  future release. Learn more about experimental features in our documentation:
  https://docs.docker.com/go/experimental/

Commands:
  annotate    Add additional information to a local image manifest
  create      Create a local manifest list for annotating and pushing to a registry
  inspect     Display an image manifest, or manifest list
  push        Push a manifest list to a repository
  rm          Delete one or more manifest lists from local storage

Run 'docker manifest COMMAND --help' for more information on a command.

Since this can be done entirely on the registry, there are other tools available that don't need to know about docker. Two of those that I know of are Crane from Google and regctl from myself:

$ crane index append --help
This sub-command pushes an index based on an (optional) base index, with appended manifests.

The platform for appended manifests is inferred from the config file or omitted if that is infeasible.

Usage:
  crane index append [flags]

Examples:
 # Append a windows hello-world image to ubuntu, push to example.com/hello-world:weird
  crane index append ubuntu -m hello-world@sha256:87b9ca29151260634b95efb84d43b05335dc3ed36cc132e2b920dd1955342d20 -t example.com/hello-world:weird

  # Create an index from scratch for etcd.
  crane index append -m registry.k8s.io/etcd-amd64:3.4.9 -m registry.k8s.io/etcd-arm64:3.4.9 -t example.com/etcd

Flags:
      --docker-empty-base   If true, empty base index will have Docker media types instead of OCI
      --flatten             If true, appending an index will append each of its children rather than the index itself (default true)
  -h, --help                help for append
  -m, --manifest strings    References to manifests to append to the base index
  -t, --tag string          Tag to apply to resulting image

Global Flags:
      --allow-nondistributable-artifacts   Allow pushing non-distributable (foreign) layers
      --insecure                           Allow image references to be fetched without TLS
      --platform platform                  Specifies the platform in the form os/arch[/variant][:osversion] (e.g. linux/amd64). (default all)
  -v, --verbose                            Enable debug logs

$ regctl index create --help
Create a manifest list or OCI Index.

Usage:
  regctl index create <image_ref> [flags]

Aliases:
  create, init, new

Flags:
      --annotation stringArray        Annotation to set on manifest
      --artifact-type string          Include an artifactType value
      --by-digest                     Push manifest by digest instead of tag
      --desc-annotation stringArray   Annotation to add to descriptors of new entries
      --desc-platform string          Platform to set in descriptors of new entries
      --digest stringArray            Digest to include in new index
      --digest-tags                   Include digest tags
      --format string                 Format output with go template syntax
  -h, --help                          help for create
  -m, --media-type string             Media-type for manifest list or OCI Index (default "application/vnd.oci.image.index.v1+json")
      --platform stringArray          Platforms to include from ref
      --ref stringArray               References to include in new index
      --referrers                     Include referrers
      --subject string                Specify a subject tag or digest (this manifest must already exist in the repo)

E.g. with regctl, that would look like:

$ regctl index create \
  --ref registry.example.org/someimage:1.0-amd64 \
  --ref registry.example.org/someimage:1.0-arm64 \
  registry.example.org/someimage:1.0