Search code examples
gitlabcontinuous-integrationcodecov

GitLab CI external test for coverage "fails"

I am using GitLab and its CI for a project.

I used to test coverage with some CI jobs until these scripts stopped working ("keyword cobertura not valid").

Simultaneously I found that the CI added some "external" jobs automatically handling coverage (see screenshot). I don't why it appeared, maybe because I have linked the project with Codecov external site. This was a pleasant surprise at the time because I didn't have to maintain a special script for coverage.

However eventually now these external coverage tests are failing and I can't merge my changes because of it. Worst part is that these are not normal scripts so I can't see what is wrong with it. And, there is no Retry button even (see screenshot, on the right).

I don't want to throw away my otherwise perfectly working merge request.

How can I see what is wrong about this part of the CI?

Clicking on the failed test send me to Codecov website and I don't see anything wrong with it.

screenshot

Here it the link to the pipeline: https://gitlab.com/correaa/boost-multi/-/pipelines/540520025

I think I solved the problem, it could have been that coverage percentage decreased (by 0.01% !) and that was interpreted by "the system" as failure. I added test to cover some uncovered lines and the problem was solved.

If this is the right interpretation, this is indeed nice, but also scary, because some big changes sometimes require a hit in coverage. In my particular example, what happened is that I simplified code and the number of total lines when down, making the covered fraction go lower than previously.

Solution

  • I think this error might have something to do with the coverage range you have declared.

    Looking at your .codecov.yml file:

    coverage:
  precision: 2
  round: down
  range: "99...100"

    You're excluding 100% when using three dots in the range, and you have achieved 100% coverage with this branch. I feel like this shouldn't matter, but you could be hitting an edge case with codecov. Maybe file a bug report with them.

    Try changing the range to 99..100. Quotes should be unnecessary.

    https://docs.codecov.com/docs/coverage-configuration