I'm having a weird issue I can't figure out. It's driving me insane.
I have a Maven project for which I'm running the same command both locally and on Jenkins. The environment is the same as far as I can tell:
Command:
mvn -Ddependency.surefire.verbose=10 -Pcoverage verify -B -U
Relevant POM section:
<build>
<pluginManagement>
<plugins>
<plugin>
<groupId>org.jacoco</groupId>
<artifactId>jacoco-maven-plugin</artifactId>
<version>0.8.10</version>
</plugin>
...
</plugins>
</pluginManagement>
...
</build>
...
<profiles>
<profile>
<id>coverage</id>
<build>
<plugins>
<plugin>
<groupId>org.jacoco</groupId>
<artifactId>jacoco-maven-plugin</artifactId>
<configuration>
<excludes>
<exclude>com/XXX/framework/dataproviders/**/*</exclude>
<exclude>com/XXX/v2/basetest/**/*</exclude>
<exclude>**/*Exception*</exclude>
</excludes>
</configuration>
<executions>
<execution>
<goals>
<goal>prepare-agent</goal>
</goals>
</execution>
<!-- attached to Maven test phase -->
<execution>
<id>report</id>
<phase>test</phase>
<goals>
<goal>report</goal>
</goals>
</execution>
<!-- Add this checking -->
<execution>
<id>jacoco-check</id>
<goals>
<goal>check</goal>
</goals>
<configuration>
<rules>
<rule>
<element>BUNDLE</element>
<limits>
<limit>
<counter>INSTRUCTION</counter>
<value>COVEREDRATIO</value>
<minimum>60%</minimum>
</limit>
<limit>
<counter>CLASS</counter>
<value>MISSEDCOUNT</value>
<maximum>0</maximum>
</limit>
</limits>
</rule>
<rule>
<element>PACKAGE</element>
<limits>
<limit>
<counter>LINE</counter>
<value>COVEREDRATIO</value>
<minimum>60%</minimum>
</limit>
</limits>
</rule>
</rules>
<skip>${skipTests}</skip>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
One of the tests that have the issue (only tests with DataProviders) shows this pattern in the logs. Enabled verbose to level 10 in hopes that something would show up.
Locally:
[2023-06-06 16:55:46.795] [INFO] [TestClass] Creating TestClass for [ClassImpl class=com.XXX.framework.utils.TimeUtilsTest]
[2023-06-06 16:55:46.796] [INFO] Method public java.lang.Object[][] com.XXX.framework.utils.TimeUtilsTest.getTimeData() has a @Test annotation but also a return value: ignoring it. Use <suite allow-return-values="true"> to fix this
[2023-06-06 16:55:46.797] [INFO] [TestClass] Adding method TimeUtilsTest.testToFormattedString(java.util.concurrent.TimeUnit,int,java.lang.String,com.XXX.framework.annotations.TestNameParameter)[pri:0, instance:null] on TestClass class com.XXX.framework.utils.TimeUtilsTest
...
[2023-06-06 16:55:47.316] [INFO] ===== Test class
com.XXX.framework.utils.TimeUtilsTest
[2023-06-06 16:55:47.316] [INFO] @Test TimeUtilsTest.testToFormattedString(java.util.concurrent.TimeUnit,int,java.lang.String,com.XXX.framework.annotations.TestNameParameter)[pri:0, instance:com.XXX.framework.utils.TimeUtilsTest@283eb984]
[2023-06-06 16:55:47.316] [INFO] ======
...
[TestNG] INVOKING: "Surefire test" - com.XXX.framework.utils.TimeUtilsTest.testToFormattedString(java.util.concurrent.TimeUnit,int,java.lang.String,com.XXX.framework.annotations.TestNameParameter)(value(s): NANOSECONDS, 60, "60ns", TestNameParameter(customName=60 NANOSECONDS))
... Test executes and results, etc.
On Jenkins:
[2023-06-06 21:02:29.270] [INFO] [TestClass] Creating TestClass for [ClassImpl class=com.XXX.framework.utils.TimeUtilsTest]
[2023-06-06 21:02:29.271] [INFO] Method public java.lang.Object[][] com.XXX.framework.utils.TimeUtilsTest.getTimeData() has a @Test annotation but also a return value: ignoring it. Use <suite allow-return-values="true"> to fix this
[2023-06-06 21:02:29.271] [INFO] [TestClass] Adding method TimeUtilsTest.testToFormattedString(java.util.concurrent.TimeUnit,int,java.lang.String,com.XXX.framework.annotations.TestNameParameter)[pri:0, instance:null] on TestClass class com.XXX.framework.utils.TimeUtilsTest
...
[2023-06-06 21:02:29.529] [INFO] ===== Test class
com.XXX.framework.utils.TimeUtilsTest
[2023-06-06 21:02:29.529] [INFO] @Test TimeUtilsTest.testToFormattedString(java.util.concurrent.TimeUnit,int,java.lang.String,com.XXX.framework.annotations.TestNameParameter)[pri:0, instance:com.XXX.framework.utils.TimeUtilsTest@52f9a620]
[2023-06-06 21:02:29.529] [INFO] ======
I see no other references about the test class in the rest of the trace. Basically, all tests with a DataProvider don't run.
Maybe someone has a suggestion on what else to look for.
I've tried the following:
The issue was a bit convoluted but neither TestNG nor Sunfire were at fault. It was a mix of a custom TestNG listener skipping the test but not logging when it did. The test was being skipped due to another test injecting a parameter in the environment that was causing the listener to skip the test.
Once the test cleaned properly the environment when done the issue went away.