Different jacoco coverage results in a github action environment

Question

is there any reason for a jacoco coverage report to produce different results when running it in a CI environment?

I have the following situation after migrating to Java 17. when I run ./gradlew clean build the report is generated and the jacocoTestCoverageVerification doesnt fail.

But when the same command is executed in a github runner I have a very different result, the coverage report is also created with different coverage values and some classes just fail the coverage verification.

Unfortunately, I cannot provide code snippets but I already tried this:

• Run the github runner on my machine, with the same jdk and it produces a different result if I just run the build in the command line :(
• add jvmargs noverify for tests tasks
• run the github action in a container instead of a self-hosted, same result
• other devs have also built the project on different machines without error
• run the same build command of the ci

So it seems that something in a github action env produces this behavior.

versions:

• jacoco plugin: 0.8.7
• gradle 7.3
• java 17

Answer 1

The problem here was lombok config files. It turns out that the new version of lombok plugin compatible with java 17 and Gradle 7.3 does not generate the lombok.config files anymore.

In the previous version, these files were generated by io.freefair.lombok plugin, and we had them in .gitignore. Because of that, a local build worked just fine, since the files were still there.

I realized this by looking at the generated reports and I noticed that lombok classes were being analyzed only by the CI.

In other cases, a similar problem could be related to this: https://www.eclemma.org/jacoco/trunk/doc/classids.html