Search code examples
gitlabcicd

Download the latest artifacts of failed gitlab pipeline

I want to automatically load test results of my gitlab pipeline (basically one xml file) into my project management application. No direct connection between these two is possible. However the gitlab API offers the following options to download artifacts of a pipeline:

  • all artifacts of the latest successful pipeline run (selected by job name)

GET /projects/:id/jobs/artifacts/:ref_name/download?job=name`

  • all artifacts of a specific job (selected by job id)

GET /projects/:id/jobs/:job_id/artifacts

  • a single artifact file (selected by job id)

GET /projects/:id/jobs/:job_id/artifacts/*artifact_path

My current situation is following:

I have test reports which are saved inside the job artifacts when running the pipeline. The artifacts are created on every run of the pipeline independent of its outcome

gitlab-ci.yaml

...
artifacts:

     when: always
...

The artifact I am trying to download has a dynamic name

./reports/junit/test-results-${CI_JOB_ID}.xml

If I now want to download the latest test results to a different server than the gitlab server, I have to realize that I don't know the latest job ID, which means:

  • I can't access the artifact directly because it has a dynamic name
  • I can't access the artifacts of a specific job
  • I can access the artifacts of the latest job, but only if it was successful

This leaves me with the situation, that i only get to download the latest test results, if nothing went wrong while testing. To put it mildly, this is suboptimal.

Is there some way to download the artifacts from the latest job run (without knowing the job ID), independent of its outcome?

Solution

  • Is there some way to download the artifacts from the latest job run (without knowing the job ID), independent of its outcome?

    In order to achieve this we will use the Gitlab API in combination with the jq package.

    Let's break down this question into components.

    Firstly, we need to find out the id of the last executed pipeline for this project. https://docs.gitlab.com/ee/api/pipelines.html#list-project-pipelines

    GET /projects/:id/pipelines

    For this call you will need your access token, if you don't have one already check https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html#create-a-personal-access-token https://docs.gitlab.com/ee/user/project/settings/project_access_tokens.html#create-a-project-access-token

    You will also need the Project ID

    enter image description here

    LAST_PIPELINE_ID=$(curl -s --header "PRIVATE-TOKEN: <access_token>" https://gitlab.com/api/v4/projects/<project_id>/pipelines | jq '.[0].id')

    Next we will retrieve the job id by providing the job name by using the following API https://docs.gitlab.com/ee/api/jobs.html#list-pipeline-jobs

    GET /projects/:id/pipelines/:pipeline_id/jobs

    In your case you need to change the following example with your job's name, in this example let's call it my_job

    JOB_ID=$(curl -s --header "PRIVATE-TOKEN: <access_token>" https://gitlab.com/api/v4/projects/<project_id>/pipelines/$LAST_PIPELINE_ID/jobs | jq '.[] | select(.name=="my_job") | .id')

    Now we are ready to actually retrieve the artifacts, with the following API

    GET /projects/:id/jobs/:job_id/artifacts

    https://docs.gitlab.com/ee/api/job_artifacts.html#get-job-artifacts

    wget -U "Mozilla/5.0 (X11; U; Linux i686; en-US) AppleWebKit/534.17 (KHTML,like Gecko) Ubuntu/11.04 Chromium/11.0.654.0 Chrome/11.0.654.0 Safari/534.17" --header "PRIVATE-TOKEN: <access_token>" "https://gitlab.com/api/v4/projects/<project_id>/jobs/$JOB_ID/artifacts" -O artifacts.zip

    The artifacts are available as artifacts.zip in the folder you executed wget from

    Combining them here for clarity

    LAST_PIPELINE_ID=$(curl -s --header "PRIVATE-TOKEN: <access_token>" https://gitlab.com/api/v4/projects/<project_id>/pipelines | jq '.[0].id')
JOB_ID=$(curl -s --header "PRIVATE-TOKEN: <access_token>" https://gitlab.com/api/v4/projects/<project_id>/pipelines/$LAST_PIPELINE_ID/jobs | jq '.[] | select(.name=="my_job") | .id')
wget -U "Mozilla/5.0 (X11; U; Linux i686; en-US) AppleWebKit/534.17 (KHTML,like Gecko) Ubuntu/11.04 Chromium/11.0.654.0 Chrome/11.0.654.0 Safari/534.17" --header "PRIVATE-TOKEN: <access_token>" "https://gitlab.com/api/v4/projects/<project_id>/jobs/$JOB_ID/artifacts" -O artifacts.zip