Search code examples
dockerdocker-registry

Private Docker Registry not Updating Images

I have a private Docker registry setup for my images. At the moment, I build 3 separate images for the application each of which has 3 platform builds (amd64, armv7 and arm64v8) for a total of 9 images being built and pushed to the registry.

I created a script to do this for me:

  1. Build the images with docker compose build
  2. Push the images with docker compose push
  3. Create each manifest for the images
  4. Push each manifest

Unfortunately, it seems that, when I make a change to an image, it does not update my registry with the new image. It says it pushed everything successfully, as far as I can tell, but when I pull the image it is still the same build from before my change.

Also note that I have confirmed that the images are completely wiped from a machine before pulling from the registry.

In order to get it to upload and ensure that I get a new image in the registry, I have to:

  1. Shutdown the registry
  2. Delete the repository with sudo rm -rf data/docker
  3. Bring the registry back up
  4. Delete all local images on my PC
  5. Delete the .docker/manifests folder on my PC (otherwise it errors when pushing)
  6. Build and push everything

I'm assuming there's a better way to do this, but I can't figure out how. It seems the comparison nature of newly built vs stored images is really lacking at least for me. Did I setup my registry wrong?

Thanks in advance for the help!

Here is my Docker Compose file:

#############################################
#        Docker Compose Build & Push        #
#############################################

version: '3.8'

services:

  node-red-amd64:
    image: app.registry.com/node-red-amd64:${NODE_RED_VERSION}
    platform: linux/amd64
    build:
      context: ..
      dockerfile: docker/node_red/Dockerfile
      args:
        - NODE_RED_VERSION=${NODE_RED_VERSION}

  node-red-arm64v8:
    image: app.registry.com/node-red-arm64v8:${NODE_RED_VERSION}
    platform: linux/arm64/v8
    build:
      context: ..
      dockerfile: docker/node_red/Dockerfile
      args:
        - NODE_RED_VERSION=${NODE_RED_VERSION}

  node-red-armv7:
    image: app.registry.com/node-red-armv7:${NODE_RED_VERSION}
    platform: linux/arm/v7
    build:
      context: ..
      dockerfile: docker/node_red/Dockerfile
      args:
        - NODE_RED_VERSION=${NODE_RED_VERSION}

  postgres-amd64:
    image: app.registry.com/postgres-amd64:${POSTGRES_VERSION}
    platform: linux/amd64
    build:
      context: ..
      dockerfile: docker/postgres/Dockerfile
      args:
        - POSTGRES_VERSION=${POSTGRES_VERSION}

  postgres-arm64v8:
    image: app.registry.com/postgres-arm64v8:${POSTGRES_VERSION}
    platform: linux/arm64/v8
    build:
      context: ..
      dockerfile: docker/postgres/Dockerfile
      args:
        - POSTGRES_VERSION=${POSTGRES_VERSION}

  postgres-armv7:
    image: app.registry.com/postgres-armv7:${POSTGRES_VERSION}
    platform: linux/arm/v7
    build:
      context: ..
      dockerfile: docker/postgres/Dockerfile
      args:
        - POSTGRES_VERSION=${POSTGRES_VERSION}

  app-amd64:
    image: app.registry.com/app-amd64:${APP_VERSION}
    platform: linux/amd64
    build:
      context: ..
      dockerfile: docker/app/Dockerfile
      args:
        - NODEJS_VERSION=${NODEJS_VERSION}

  app-arm64v8:
    image: app.registry.com/app-arm64v8:${APP_VERSION}
    platform: linux/arm64/v8
    build:
      context: ..
      dockerfile: docker/app/Dockerfile
      args:
        - NODEJS_VERSION=${NODEJS_VERSION}

  app-armv7:
    image: app.registry.com/app-armv7:${APP_VERSION}
    platform: linux/arm/v7
    build:
      context: ..
      dockerfile: docker/app/Dockerfile
      args:
        - NODEJS_VERSION=${NODEJS_VERSION}

And my script to build and push everything:

#!/bin/bash
# shellcheck disable=SC2046
#
# This script is intended to deploy all Docker images to the app.registry.com registry
# 1. Login to the Docker registry
# 2. Build all images with Docker Compose
# 3. Push all images to the Docker registry
# 4. Create manifests for each arch build of the images
# 5. Push the manifests to the Docker registry
#
# By using manifests we can pull a single image (e.g. app.registry.com/app:latest)
# and get the one that is applicable to the current CPU architecture we're using
#
# Versions are all specified in the .env file and populated throughout the Docker files.
# The only limitation is that any new builds or architectures that are added to
# docker-compose.yml must also be repeated in this file as a manifest entry (see line 35).
#
# Export all the environment variables that are present in .env
echo "Exporting all variables from .env"
export $(grep -v '^#' .env | xargs -d '\n')

# Login to the registry
echo "Logging in to https://app.registry.com Docker registry"
docker login https://app.registry.com

# Build the images
echo "Building images with Docker Compose"
docker-compose build

# Push the images
echo "Pushing images to the Docker registry"
docker-compose push

# Create Docker manifests
# Note that all the images created in the docker-compose.yml
# should also be listed here
echo "Creating NodeRed v${NODE_RED_VERSION} manifest"
docker manifest create -a \
 app.registry.com/node-red:"${NODE_RED_VERSION}" \
 app.registry.com/node-red-amd64:"${NODE_RED_VERSION}" \
 app.registry.com/node-red-arm64v8:"${NODE_RED_VERSION}" \
 app.registry.com/node-red-armv7:"${NODE_RED_VERSION}"

echo "Creating Postgres v${POSTGRES_VERSION} manifest"
docker manifest create -a \
 app.registry.com/postgres:"${POSTGRES_VERSION}" \
 app.registry.com/postgres-amd64:"${POSTGRES_VERSION}" \
 app.registry.com/postgres-arm64v8:"${POSTGRES_VERSION}" \
 app.registry.com/postgres-armv7:"${POSTGRES_VERSION}"

echo "Creating APP v${APP_VERSION} manifest"
docker manifest create -a \
 app.registry.com/app:"${APP_VERSION}" \
 app.registry.com/app-amd64:"${APP_VERSION}" \
 app.registry.com/app-arm64v8:"${APP_VERSION}" \
 app.registry.com/app-armv7:"${APP_VERSION}"

# Push Docker manifests
echo "Pushing NodeRed v${NODE_RED_VERSION} manifest"
docker manifest push app.registry.com/node-red:"${NODE_RED_VERSION}"

echo "Pushing Postgres v${POSTGRES_VERSION} manifest"
docker manifest push app.registry.com/postgres:"${POSTGRES_VERSION}"

echo "Pushing APP v${APP_VERSION} manifest"
docker manifest push app.registry.com/app:"${APP_VERSION}"

EDIT:

Here's my docker-compose.yml for the registry itself. This uses Nginx for access restriction:

version: '3.8'

services:
  registry:
    restart: always
    image: registry:2
    container_name: registry
    ports:
    - '5000:5000'
    environment:
      REGISTRY_AUTH: htpasswd
      REGISTRY_AUTH_HTPASSWD_REALM: Registry
      REGISTRY_AUTH_HTPASSWD_PATH: /auth/registry.password
      REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY: /data
    volumes:
      - ./auth:/auth
      - ./data:/data
Solution

  • My guess is docker manifest create isn't doing what you want. Since you're amending an existing manifest list, that may be a noop when the platform already exists, or it could be adding more and more entries to the list and runtimes pick from the beginning of the list as they should. To debug, I'd inspect the manifest:

    docker manifest inspect "app.registry.com/node-red:${NODE_RED_VERSION}"

    You can try docker manifest rm "app.registry.com/node-red:${NODE_RED_VERSION}" at the beginning (or end) of your script to reset (or cleanup) the local storage.

    However, I'd opt for buildx for this scenario. It's GA so you aren't relying on experimental features, and it wraps all the steps into one command, e.g.:

    docker buildx build \
  --build-arg "NODE_RED_VERSION=${NODE_RED_VERSION}" \
  --platform linux/amd64,linux/arm64,linux/arm/v7 \
  -f docker/node_red/Dockerfile \
  -t "app.registry.com/node-red:${NODE_RED_VERSION}" --push \
  ..