Search code examples
gitlabversion-controlgitlab-cicicd

Using GitLab CI CD artifacts from previous job in next job from same stage - where are the artifacts downloaded to?

I have the following two stages in my GitLab pipeline:

  • build-base - builds a base image (only if a commit is pushed that changes certain files)
  • build - consists of two jobs:
    • build-deps - adds and builds some dependencies
    • build - builds a final image that is derived from the base one and also "installs" the artifacts generated by build-deps job.

The build job looks similar to:

build-deps:
  stage: build
  tags:
    - runnerX
  artifacts:
    untracked: true
  script:
    # Fix paths for shell runner
    - export PATH=$PATH:...
    - export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:...
    # Run the tools to build the dependencies
    - ...

The build job looks similar to:

build:
  stage: build
  image:
    name: gcr.io/kaniko-project/executor:debug
    entrypoint: ['']
  tags:
    - runnerY
  needs:
    - job: build-deps
      artifacts: true
  script:
    - echo "{\"auths\":{\"${CI_REGISTRY}\":{\"auth\":\"$(printf "%s:%s" "${CI_REGISTRY_USER}" "${CI_JOB_TOKEN}" | base64 | tr -d '\n')\"}}}" > /kaniko/.docker/config.json
    - /kaniko/executor
       --context ./final
       --dockerfile Dockerfile
       --insecure
       --skip-tls-verify
       --skip-tls-verify-pull
       --insecure-pull
       --destination "${CI_REGISTRY_IMAGE}/result:latest"

In the command line output in the GitLab web UI of the build job I can see

Downloading artifacts 00:01
Downloading artifacts for build-deps (4968461)...
Downloading artifacts from coordinator... ok        host=example.com id=4968461 responseStatus=200 OK token=XXXXXXXXX

In order to include the files in my Dockerfile that is used to create the final image (dependencies plus main application) I need to copy the dependencies whereever those were downloaded to. Sadly I cannot find the location.

I tried ls and find (recursion included) inside the script section of the job but I cannot find neither a ZIP file (the default in the Artifacts web UI generates an artifacts.zip) nor the contents of that ZIP file.

Note that I do use different runners (due to the requirements of the different jobs) if that counts for something.

Solution

  • The problem was the context, which I give as argument to Kaniko (in a similar fashion podman, docker etc. will also be affected).

    I changed to Kaniko call to the following:

    - /kaniko/executor
   --context .
   --dockerfile ./final/Dockerfile
   --insecure
   --skip-tls-verify
   --skip-tls-verify-pull
   --insecure-pull
   --destination "${CI_REGISTRY_IMAGE}/result:latest"

    I forgot that the --context that provides the building environment can be different (as I have done here) to where the --dockerfile is pointing at. That is why the building step was failing because I was trying to COPY something from ./final (where my Dockerfile is located) and not from the actual directory where the checkout is done.