docker azure-devops azure-pipelines azure-pipelines-yaml azure-pipelines-tasks

Work around for Docker layer caching not working in Azure Pipeline

I'm hoping to cache a specific stage in my multi-stage Dockerfile that my test stages use in an effort speed the build process up. Otherwise, it builds for unit testing and then for integration testing.

Here is a bare bones example of one of the Dockerfile:

# creating a node base
FROM node:16-slim as node-base
ENV CI=true


# builder-base is used to build dependencies
FROM node-base as builder-base
COPY ./package-lock.json ./package.json ./
RUN npm ci --production


# 'development' stage installs all dev deps and can be used to develop code.
FROM builder-base as development
WORKDIR /app
COPY . . 
RUN npm ci
EXPOSE 4001
CMD ["npm", "start"]


# 'unit-tests' stage 
FROM development AS unit-tests
RUN npm test -- --coverage --testNamePattern=UT: 


# 'integration-tests' stage 
FROM development AS integration-tests
RUN npm test -- --coverage --testNamePattern=IT:

I'd like to cache the development stage, pull it and just run the unit-tests and integration-tests stages without out building development twice.

I found this question which I'm trying to implement:

How to Enable Docker layer caching in Azure DevOps

The top answer there is this:

- task: Docker@2
  inputs:
    containerRegistry: '$(ContainerRegistryName)'
    command: 'login'

- script: "docker pull $(ACR_ADDRESS)/$(REPOSITORY):latest"
  displayName: Pull latest for layer caching
  continueOnError: true # for first build, no cache

- task: Docker@2
  displayName: build
  inputs:
    containerRegistry: '$(ContainerRegistryName)'
    repository: '$(REPOSITORY)'
    command: 'build'
    Dockerfile: './dockerfile '
    buildContext: '$(BUILDCONTEXT)'
    arguments: '--cache-from=$(ACR_ADDRESS)/$(REPOSITORY):latest' 
    tags: |
      $(Build.BuildNumber)
      latest

- task: Docker@2
  displayName: "push"
  inputs:
    command: push
    containerRegistry: "$(ContainerRegistryName)"
    repository: $(REPOSITORY) 
    tags: |
      $(Build.BuildNumber)
      latest

I've repurposed it for my pipeline like so:

# pr.yaml
# # This is triggered by the PR and branch policies
trigger: none

# Specify this to run on the app repo
resources:
  repositories:
  - repository:app
    type: git
    name: app

# Read in the base variable template
variables:
- template: templates/variables.yaml

# Use the ubuntu-latest image
pool:
  vmIMage: $(vmImageName)

# Stages and their templates for the PR pipeline
stages:
# Checks to see what services in the mono repo have changed by comparing
# the PR code to trunk
- template: templates/changed.yaml
  parameters:
    comparedTo: origin/trunk

# Run unit tests for each changed service
- template: templates/services.yaml
  parameters:
    stageName: BuildDev
    stageDisplayName: Build dev stage for services...
    dockerCommand: build
    phrase: build dev
    target: development
    tag: latest

# Run unit tests for each changed service
- template: templates/services.yaml
  parameters:
    stageName: UnitTests
    stageDisplayName: Run unit tests for services...
    dockerCommand: build
    phrase: unit test
    target: unit-tests
    tag: ut-$(Build.BuildNumber)

# Run integration tests for each changed service
- template: templates/services.yaml
  parameters:
    stageName: IntegrationTests
    stageDisplayName: Run integration tests for services...
    dockerCommand: build
    phrase: integration test
    target: integration-tests
    tag: it-$(Build.BuildNumber)

# services.yaml
parameters:
- name: stageName
  default: ''
- name: stageDisplayName
  default: ''
- name: phrase
  default: ''
- name: dockerCommand
  default: ''
- name: target
  default: ''
- name: tag
  default: ''
- name: services
  type: object
  default:
  - admin-v2
  - api
  - portal

stages:
- stage: ${{ parameters.stageName }}
  displayName: ${{ parameters.stageDisplayName }}
  # Run if detectChanges ran successfully
  dependsOn: 
  - Changed  
  - ${{ if eq(parameters.stageName, 'UnitTests') }}:
    - BuildDev
  - ${{ if eq(parameters.stageName, 'IntegrationTests') }}:
    - BuildDev
    - UnitTests
  condition: succeeded()
  jobs:
  # Runs for all other stages

  - ${{ each service in parameters.services }}:
    - template: docker.yaml
      parameters:
        service: ${{ service }}
        stageName: ${{ parameters.stageName }}
        jobName: ${{ service }}${{ parameters.stageName }}
        jobDisplayName: Run ${{ parameters.phrase }} for ${{ service }} service...
        taskDisplayName: Run ${{ service }} ${{ parameters.phrase }} tasks...
        dockerCommand: ${{ parameters.dockerCommand }}
        target: ${{ parameters.target }}
        tag: ${{ parameters.tag }}

# docker.yaml
parameters:
- name: stageName
  default: ''
- name: service
  default: ''
- name: jobName
  default: ''
- name: jobDisplayName
  default: ''
- name: taskDisplayName
  default: ''
- name: dockerCommand
  default: ''
- name: target
  default: ''
- name: tag
  default: ''

jobs:
- job: 
  displayName: ${{ parameters.jobDisplayName }}
  # Handle whether to run for service or not
  variables:
    servicesChanged: $[ stageDependencies.Changed.Changes.outputs['detectChanges.servicesChanged'] ]
  condition: or(contains(variables['servicesChanged'], '${{ parameters.service }}'), eq(variables['Build.Reason'], 'Manual'))
  steps: 
  # Set to app repo
  - checkout: app

  # Create mysecrets.txt primarily for Django system check
  - bash: |
      printenv >> $(dockerFilePath)/${{ parameters.service }}/mysecrets.txt
    displayName: Create mysecrets.txt for ${{ parameters.service }}
    env:
      DJANGO_SECRET_KEY: $(DJANGO_SECRET_KEY)

  - ${{ if not(eq(parameters.stageName, 'BuildDev')) }}:
    # Run the Docker task
    - task: Docker@2
      inputs:
        containerRegistry: $(dockerRegistryServiceConnection)
        command: login

    - script: docker pull $(containerRegistry)/$(imageRepository)-${{ parameters.service }}:latest
      displayName: Pull latest for layer caching
      continueOnError: true # for first build, no cache

    - task: Docker@2
      # Run if there have been changes
      displayName: ${{ parameters.taskDisplayName }}
      inputs:
        command: ${{ parameters.dockerCommand }}
        repository: $(imageRepository)-${{ parameters.service }}
        dockerfile: $(dockerFilePath)/${{ parameters.service }}/docker/Dockerfile
        buildContext: $(dockerFilePath)/${{ parameters.service }}
        containerRegistry: $(dockerRegistryServiceConnection)
        arguments: |
          --target ${{ parameters.target }} 
          --cache-from=$(containerRegistry)/$(imageRepository)-${{ parameters.service }}:latest
        tags: |
          ${{ parameters.tag }}-$(Build.BuildNumber)
      env:
        DOCKER_BUILDKIT: 1

  - ${{ if eq(parameters.stageName, 'BuildDev') }}:
    - task: Docker@2
      # Run if there have been changes
      displayName: ${{ parameters.taskDisplayName }}
      inputs:
        command: ${{ parameters.dockerCommand }}
        repository: $(imageRepository)-${{ parameters.service }}
        dockerfile: $(dockerFilePath)/${{ parameters.service }}/docker/Dockerfile
        buildContext: $(dockerFilePath)/${{ parameters.service }}
        containerRegistry: $(dockerRegistryServiceConnection)
        arguments: --target ${{ parameters.target }}
        tags: |
          ${{ parameters.tag }}
      env:
        DOCKER_BUILDKIT: 1
    - task: Docker@2
      displayName: Pushing ${{ parameters.service }} ${{ parameters.tag }} to ACR
      inputs:
        command: push
        repository: $(imageRepository)-${{ parameters.service }}      
        containerRegistry: $(dockerRegistryServiceConnection)
        tags: |
          ${{ parameters.tag }}

The stageName BuildDev is where the development stage would be built.

Everything runs successfully. The task where the image is pulled shows it being pulled, then task where --target unit-tests runs shows importing cache manifest from ***/app-admin:dev-20211030.11, but it still builds the development and preceeding stages. It pulls it, sees it's there, and decides to build it anyways.

Here are those logs:

# Pull Job
2021-11-02T23:29:33.0494768Z ##[section]Starting: Pull latest for layer caching
2021-11-02T23:29:33.0502467Z ==============================================================================
2021-11-02T23:29:33.0502844Z Task         : Command line
2021-11-02T23:29:33.0503204Z Description  : Run a command line script using Bash on Linux and macOS and cmd.exe on Windows
2021-11-02T23:29:33.0503544Z Version      : 2.182.0
2021-11-02T23:29:33.0504030Z Author       : Microsoft Corporation
2021-11-02T23:29:33.0504416Z Help         : https://learn.microsoft.com/azure/devops/pipelines/tasks/utility/command-line
2021-11-02T23:29:33.0504832Z ==============================================================================
2021-11-02T23:29:33.2083789Z Generating script.
2021-11-02T23:29:33.2098563Z Script contents:
2021-11-02T23:29:33.2099499Z docker pull ***/app-admin-v2:latest
2021-11-02T23:29:33.2100315Z ========================== Starting Command Output ===========================
2021-11-02T23:29:33.2155259Z [command]/usr/bin/bash --noprofile --norc /home/vsts/work/_temp/5e7263fa-2853-4e6d-b303-62fe80cfacdc.sh
2021-11-02T23:29:34.5925241Z latest: Pulling from app-admin-v2
2021-11-02T23:29:34.5933140Z b380bbd43752: Pulling fs layer
2021-11-02T23:29:34.5933539Z 8d36a6ce056a: Pulling fs layer
2021-11-02T23:29:34.5933881Z f54546b42be1: Pulling fs layer
2021-11-02T23:29:34.5934203Z f5bd69d20a35: Pulling fs layer
2021-11-02T23:29:34.5934568Z 21494383f180: Pulling fs layer
2021-11-02T23:29:34.5934902Z 87500a3a7192: Pulling fs layer
2021-11-02T23:29:34.5935238Z debc4a9f3725: Pulling fs layer
2021-11-02T23:29:34.5935558Z 1b67e176d924: Pulling fs layer
2021-11-02T23:29:34.5935890Z d603a960b591: Pulling fs layer
2021-11-02T23:29:34.5936223Z 9e85221572ee: Pulling fs layer
2021-11-02T23:29:34.5943277Z f5bd69d20a35: Waiting
2021-11-02T23:29:34.5943992Z 21494383f180: Waiting
2021-11-02T23:29:34.5944321Z 87500a3a7192: Waiting
2021-11-02T23:29:34.5944642Z debc4a9f3725: Waiting
2021-11-02T23:29:34.5944958Z 1b67e176d924: Waiting
2021-11-02T23:29:34.5945246Z d603a960b591: Waiting
2021-11-02T23:29:34.5945573Z 9e85221572ee: Waiting
2021-11-02T23:29:34.9794967Z 8d36a6ce056a: Verifying Checksum
2021-11-02T23:29:34.9799329Z 8d36a6ce056a: Download complete
2021-11-02T23:29:35.6335949Z f5bd69d20a35: Verifying Checksum
2021-11-02T23:29:35.6337075Z f5bd69d20a35: Download complete
2021-11-02T23:29:35.8084539Z b380bbd43752: Verifying Checksum
2021-11-02T23:29:35.8120185Z b380bbd43752: Download complete
2021-11-02T23:29:35.9459756Z 21494383f180: Verifying Checksum
2021-11-02T23:29:35.9460303Z 21494383f180: Download complete
2021-11-02T23:29:35.9926957Z f54546b42be1: Verifying Checksum
2021-11-02T23:29:35.9927391Z f54546b42be1: Download complete
2021-11-02T23:29:36.1820048Z 87500a3a7192: Verifying Checksum
2021-11-02T23:29:36.1820481Z 87500a3a7192: Download complete
2021-11-02T23:29:36.4513965Z 1b67e176d924: Verifying Checksum
2021-11-02T23:29:36.4514451Z 1b67e176d924: Download complete
2021-11-02T23:29:37.1143768Z d603a960b591: Verifying Checksum
2021-11-02T23:29:37.1144264Z d603a960b591: Download complete
2021-11-02T23:29:37.3920871Z b380bbd43752: Pull complete
2021-11-02T23:29:38.0559222Z 9e85221572ee: Verifying Checksum
2021-11-02T23:29:38.0559774Z 9e85221572ee: Download complete
2021-11-02T23:29:38.5139277Z debc4a9f3725: Verifying Checksum
2021-11-02T23:29:38.5140203Z debc4a9f3725: Download complete
2021-11-02T23:29:39.0212051Z 8d36a6ce056a: Pull complete
2021-11-02T23:29:40.9828384Z f54546b42be1: Pull complete
2021-11-02T23:29:41.1410341Z f5bd69d20a35: Pull complete
2021-11-02T23:29:41.2067833Z 21494383f180: Pull complete
2021-11-02T23:29:41.2833611Z 87500a3a7192: Pull complete
2021-11-02T23:29:54.5480084Z debc4a9f3725: Pull complete
2021-11-02T23:29:54.6097840Z 1b67e176d924: Pull complete
2021-11-02T23:29:54.6823771Z d603a960b591: Pull complete
2021-11-02T23:30:04.3756447Z 9e85221572ee: Pull complete
2021-11-02T23:30:04.3801963Z Digest: sha256:64308db1d461a2aff0deaf31b5bb5694becfb2298f0c474366d1d9b695b0a441
2021-11-02T23:30:04.3836112Z Status: Downloaded newer image for ***/app-admin-v2:latest
2021-11-02T23:30:04.3874744Z ***/app-admin-v2:latest
2021-11-02T23:30:04.4026099Z ##[section]Finishing: Pull latest for layer caching

# Build Job
2021-11-02T23:30:08.6626317Z [command]/usr/bin/docker build -f /home/vsts/work/1/s/admin-v2/docker/Dockerfile --label com.azure.dev.image.system.teamfoundationcollectionuri=https://dev.azure.com/thecompany/ --label com.azure.dev.image.system.teamproject=-dev --label com.azure.dev.image.build.repository.name=production-resources --label com.azure.dev.image.build.sourceversion=809969a7bb0880a135c935c5d66ea0e2bba2c65e --label com.azure.dev.image.build.repository.uri=https://[email protected]/thecompany/-dev/_git/production-resources --label com.azure.dev.image.build.sourcebranchname=main --label com.azure.dev.image.build.definitionname= App PR --label com.azure.dev.image.build.buildnumber=20211102.3 --label com.azure.dev.image.build.builduri=vstfs:///Build/Build/1322 --label image.base.ref.name=nginx --label image.base.digest=sha256:644a70516a26004c97d0d85c7fe1d0c3a67ea8ab7ddf4aff193d9f301670cf36 --target unit-tests --cache-from=***/app-admin-v2:latest -t ***/app-admin-v2:ut-20211102.3-20211102.3 /home/vsts/work/1/s/admin-v2
2021-11-02T23:30:08.9603622Z #1 [internal] load build definition from Dockerfile
2021-11-02T23:30:08.9604133Z #1 sha256:7818e7e1291667e1af9ce6f8a463d74e3df7b64001596dd00b927ccc12c37515
2021-11-02T23:30:08.9604573Z #1 transferring dockerfile: 1.05kB done
2021-11-02T23:30:08.9604904Z #1 DONE 0.0s
2021-11-02T23:30:08.9605027Z 
2021-11-02T23:30:08.9605303Z #2 [internal] load .dockerignore
2021-11-02T23:30:08.9605713Z #2 sha256:1c18692a923cc3e70a98431aefc897940bf0c36edf7ae0f3b0b525b4d753b7fb
2021-11-02T23:30:08.9606117Z #2 transferring context: 329B done
2021-11-02T23:30:08.9607436Z #2 DONE 0.0s
2021-11-02T23:30:08.9607554Z 
2021-11-02T23:30:08.9608532Z #3 [internal] load metadata for docker.io/library/node:16-slim
2021-11-02T23:30:08.9609018Z #3 sha256:faa605aa367b596b57bbdc1bdcccade69c92d97d03d44e595a34c7e28b8d594e
2021-11-02T23:30:10.1243750Z #3 DONE 1.2s
2021-11-02T23:30:10.1243963Z 
2021-11-02T23:30:10.1245311Z #12 importing cache manifest from ***/app-admin-v2:latest
2021-11-02T23:30:10.1245833Z #12 sha256:91a404777043fddf396dcad59a4fd8b976e0224c77d860e00947ec3630b83eaf
2021-11-02T23:30:10.1246215Z #12 DONE 0.0s
2021-11-02T23:30:10.1246341Z 
2021-11-02T23:30:10.1246618Z #4 [internal] load build context
2021-11-02T23:30:10.1247014Z #4 sha256:e1bc60dd1feb9ca2db3a89f6b76ec616e7b56215986652ad691e1fc4c108a5aa
2021-11-02T23:30:10.1247447Z #4 transferring context: 713.43kB 0.0s done
2021-11-02T23:30:10.1247777Z #4 DONE 0.0s
2021-11-02T23:30:10.1247897Z 
2021-11-02T23:30:10.1254657Z #11 [node-base 1/1] FROM docker.io/library/node:16-slim@sha256:9ec1ff69c844f2de3a6a2180cd49ca75797d9f2a0fc52bb33c8a672fd0fe7e18
2021-11-02T23:30:10.1255382Z #11 sha256:af5e5b9d07d96a94506820483ad64d714f1bf9a0e5ae75b7b4e7265902c9f941
2021-11-02T23:30:10.1256263Z #11 resolve docker.io/library/node:16-slim@sha256:9ec1ff69c844f2de3a6a2180cd49ca75797d9f2a0fc52bb33c8a672fd0fe7e18 done
2021-11-02T23:30:10.1256894Z #11 sha256:9ec1ff69c844f2de3a6a2180cd49ca75797d9f2a0fc52bb33c8a672fd0fe7e18 1.21kB / 1.21kB done
2021-11-02T23:30:10.1257454Z #11 sha256:ed230d53c9d9820caa9b1bea418c1f835d15ec4d1253160e908ff31fe074ac35 1.37kB / 1.37kB done
2021-11-02T23:30:10.1258009Z #11 sha256:dd74f260f56dccc771f512ef5b2a81345e3bcefcac34c248459da36169be36b2 6.89kB / 6.89kB done
2021-11-02T23:30:10.1258404Z #11 DONE 0.1s
2021-11-02T23:30:10.2748114Z 
2021-11-02T23:30:10.2749533Z #5 [builder-base 1/2] COPY ./package-lock.json ./package.json ./
2021-11-02T23:30:10.2774757Z #5 sha256:e5803272aeaf38a29bf5ca34e14cc0c613c673044fdab9a2c27d837bb2de837c
2021-11-02T23:30:10.2775425Z #5 DONE 0.0s
2021-11-02T23:30:10.2775692Z 
2021-11-02T23:30:10.2776552Z #6 [builder-base 2/2] RUN npm ci --production
2021-11-02T23:30:10.2777798Z #6 sha256:30a22a0c38f1eebba7d10ab9f4cd8f24ce6a4599827350e6b42f38836af226ab
2021-11-02T23:30:12.6879927Z #6 2.477 npm WARN old lockfile 
2021-11-02T23:30:12.6881199Z #6 2.478 npm WARN old lockfile The package-lock.json file was created with an old version of npm,
2021-11-02T23:30:12.6881732Z #6 2.478 npm WARN old lockfile so supplemental metadata must be fetched from the registry.
2021-11-02T23:30:12.6882137Z #6 2.478 npm WARN old lockfile 
2021-11-02T23:30:12.6883737Z #6 2.479 npm WARN old lockfile This is a one-time fix-up, please be patient...
2021-11-02T23:30:12.6884195Z #6 2.479 npm WARN old lockfile 
...

Any ideas why this might be happening and how to prevent it? Really is slowing things down.

Solution

I managed to get this working after asking a related question here:

Work around for Docker layer caching not working in Azure Pipeline