Append pytest coverage to file in Gitlab CI artifacts

Question

I am trying to split my pytests in a gitlab stage to reduce the time it takes to run them. However, I am having difficulties getting the full coverage report. I am unable to use pytest-xdist or pytest-parallel due to the way our database is set up.

Build:
  stage: Build
  script:
    - *setup-build
    - *build
    - touch banana.xml   # where I write the code coverage collected by pytest-cov
    - *push
  artifacts:
    paths:
    - banana.xml
    reports:
      cobertura: banana.xml

Unit Test:
  stage: Test
  script:
    - *setup-build
    - *pull
    - docker-compose run $DOCKER_IMG_NAME pytest -m unit --cov-report xml:banana.xml --cov=app --cov-append;
  needs:
    - Build

Validity Test:
  stage: Test
  script:
    - *setup-build
    - *pull
    - docker-compose run $DOCKER_IMG_NAME pytest -m validity --cov-report xml:banana.xml --cov=app --cov-append;
  needs:
    - Build

After these two stages run (Build - 1 job, Test - 2 jobs), I go to download the banana.xml file from Gitlab but there's nothing in it, even though the jobs say Coverage XML written to file banana.xml

Am I missing something with how to get the total coverage written to an artifact file when splitting up marked tests in a gitlab pipeline stage?

Answer 1

If you want to combine the coverage reports of several different jobs, you will have to add another stage which will run after your tests. Here is a working example for me :

# You need to define the Test stage before the Coverage stage
stages:
  - Test
  - Coverage

# Your first test job
unit_test:
  stage: Test
  script:
    - COVERAGE_FILE=.coverage.unit coverage run --rcfile=.coveragerc -m pytest ./unit
  artifacts:
    paths:
      - .coverage.unit

# Your second test job which will run in parallel
validity_test:
  stage: Test
  script:
    - COVERAGE_FILE=.coverage.validity coverage run --rcfile=.coveragerc -m pytest ./validity
  artifacts:
    paths:
      - .coverage.validity

# Your coverage job, which will combine the coverage data from the two tests jobs and generate a report
coverage:
  stage: Coverage
  script:
    - coverage combine --rcfile=.coveragerc
    - coverage report
    - coverage xml -o coverage.xml
  coverage: '/\d+\%\s*$/'
  artifacts:
    reports:
      cobertura: coverage.xml

You also need to create a .coveragerc file in your repository with the following content, to specify that coverage.py needs to use relative file paths, because your tests were run on different gitlab runners, so their full path don't match :

[run]
  relative_files = True
  source =
  ./

Note : In your case it's better to use the coverage command directly (so coverage run -m pytest instead of pytest) because it provides more options, and it's what pytest uses under the hood anyway.

---

The issue in your file is that you start with creating an empty file, try to generate a report from that (which won't generate anything since the file is empty), and then pass it over to both test jobs, which both overwrite it with their local coverage report separately, and then never use it.

You need to do it the other way round, as shown in my example : run the tests first, and in a later stage, get both the test coverage data, and generate a report from that.