[Edit on Sep 20, 2023] The issue was found within the same pipeline run. It's not caused by Microsoft-hosted agent not retaining any data between different runs.
[Original Question] We have an Azure DevOps yaml pipeline running on a self-hosted agent, it works fine. When we switch to use the Microsoft-hosted agent, we got an issue retrieving the intermediate unit test docker image generated in the previous build stage.
Dockerfile
# Phase 1: Build the source code
FROM nginx:latest AS build
WORKDIR /src
RUN echo "Build completed" > build.txt
# Phase 2: Run unit tests
FROM build as testrunner
WORKDIR /testresults
RUN echo "Placeholder" > placeholder.txt
RUN echo "Unit test completed" > testresults.txt
LABEL unittestlayer=true
# Phase 3: Publish the binary
FROM build as publish
WORKDIR /publish
COPY --from=build /src/build.txt .
RUN echo "Binary publish completed" > publish.txt
# Phase 4: Build the final docker image
FROM nginx:latest AS base
WORKDIR /app
COPY --from=publish /publish .
COPY --from=testrunner /testresults/placeholder.txt .
azure-pipelines.yml
trigger:
- main
pool:
vmImage: "ubuntu-22.04"
steps:
- task: DockerCompose@0
displayName: "Build and Test"
inputs:
dockerComposeFile: "**/docker-compose.yml"
dockerComposeFileArgs: |
TAG=$(Build.BuildNumber)
projectName: "demo"
action: "Build services"
- script: |
docker images -a
export unittestlayerid=$(docker images --filter "label=unittestlayer=true" -q)
docker create --name unittestcontainer $unittestlayerid
docker cp unittestcontainer:/testresults/ $(Build.ArtifactStagingDirectory)/testresults/
docker stop unittestcontainer
docker rm unittestcontainer
displayName: "Retrieve Test and Coverage Results"
continueOnError: false
The script in the pipeline failed to get the unit test image, docker images --filter "label=unittestlayer=true" -q returned empty result. There is also no intermediate docker images in the output of docker images -a command.
We prepared the minimal example code to replicate the issue and uploaded it to a public Azure DevOps project https://dev.azure.com/liangzugeng/_git/DockerBuildPipelineRepro, and the previous failed pipeline run contains the build output https://dev.azure.com/liangzugeng/DockerBuildPipelineRepro/_build/results?buildId=59&view=logs&j=12f1170f-54f2-53f3-20dd-22fc7dff55f9
What does Microsoft-hosted agent do differently to cause the issue and how can the intermediate docker image be retrieved in latter build steps within the same build pipeline?
The cause of the issue was that the docker BuildKit engine only exports the final image, it removes all intermediate images. This is a by-design behavior.
When we migrated to the Microsoft-hosted agent, the docker build engine was changed from the classical one we had on our private agent to the new BuildKit engine. Which caused the issue.
See GitHub issues below, our solution was to build needed intermediate images in separate steps and tag them, then use the tagged intermediate images in the following build steps.
Revised azure pipeline:
trigger:
- main
pool:
vmImage: "ubuntu-22.04"
steps:
- script: |
docker build -t dockerissue.repro:build --target build .
docker build -t dockerissue.repro:testrunner --target testrunner .
docker build -t dockerissue.repro:$(Build.BuildNumber) .
displayName: "Build services"
continueOnError: false
- script: |
docker images -a
export unittestlayerid=$(docker images --filter "label=unittestlayer=true" -q)
docker create --name unittestcontainer $unittestlayerid
docker cp unittestcontainer:/testresults/ $(Build.ArtifactStagingDirectory)/testresults/
docker stop unittestcontainer
docker rm unittestcontainer
displayName: "Retrieve Test and Coverage Results"
continueOnError: false