Split build and push phases fail sometimes

Question

I have 4 gitlab-ci.yml files very similar to each other, the do this:

build database:
  stage: build
  retry: 2
  script:
    - echo 'Building the database image...'
    - docker pull ${IMAGE_PREFIX}/db:latest || true
    - docker build
        --file database/docker/Dockerfile
        --build-arg IMAGE_REGISTRY=${IMAGE_REGISTRY}
        --pull
        --cache-from ${IMAGE_PREFIX}/db:latest
        --tag ${IMAGE_PREFIX}/db:${CI_COMMIT_SHA}
        .
  rules:
    - if : '$CI_COMMIT_BRANCH == "main"'

push database:
  stage: push
  retry: 2
  script:
    - docker push ${IMAGE_PREFIX}/db:${CI_COMMIT_SHA}
    - docker rmi  ${IMAGE_PREFIX}/db:latest || true
    - docker rmi  ${IMAGE_PREFIX}/db:${CI_COMMIT_SHA}
  rules:
    - if : '$CI_COMMIT_BRANCH == "main"'

In the Gitlab Pipeline, they all pass the build stage, but the push stage seems to randomly fail with an error tag does not exist what can be happening?

I have done - docker images to debug, and indeed, when it fails, I see the image has the incorrect tag:

$ docker images
REPOSITORY                TAG                                        IMAGE ID       CREATED          SIZE
db                        d5490eeb56483b580c06f1d7a028c9a39d28eff4   a8955a972823
$ echo ${IMAGE_PREFIX}/db:${CI_COMMIT_SHA}
db:c9eaa8109b4d79d012244fccebd69c964e1469b3

I have checked and the build stage used the c9eaa8109b4d79d012244fccebd69c964e1469b3 tag. What can be happening? Why is it a random fail?

And I want to push

Answer 1

Jobs are not going to run in the same VM, although GitLab may reuse the same machine, but that's not something you can control.

The times you've found this scenario working is because it may have happened that both jobs have been allocated to the same machine.

If you want to build and push you should do it in the same step, it won't make any difference from your actual setup.

There's more documentation on how GitLab runners work in the official docs.