Search code examples
spring-bootjacocojunit5jacoco-maven-plugin

Jacoco + JUnit 5.0 DynamicTest not working

I am trying to generate a code coverage report using Jacoco with JUnit 5 and Spring Boot. And I"m trying to use the DynamicTest feature of JUnit 5. It runs successfully but the Dynamic Tests are not covered in the test coverage report generated by jacoco. Is JUnit 5 Dynamic Test not covered by Jacoco at the moment?

Here is the code:

    @RunWith(JUnitPlatform.class)
    @SelectPackages("com.troll.jpt.abc.model")
    @SelectClasses({Status.class})
    public class DynamicModelTester {

    private Status status;

    @BeforeEach
    public void setUp() {
        status = new Status();
    }

    @TestFactory
    public Stream<DynamicTest> checkDynamicTestsFromStream() {

        List<String> input = Arrays.asList("abc");
        List<String> output = Arrays.asList("abc");

        status.setCode(input.get(0));

        return input.stream().map(str ->  DynamicTest.dynamicTest("status test", () -> {
            assertEquals(output.get(0), status.getCode());
        }));
    }
}

The jacoco plugin I'm using is as :

    <plugin>
        <groupId>org.jacoco</groupId>
        <artifactId>jacoco-maven-plugin</artifactId>
        <version>0.8.2-SNAPSHOT</version>
        <executions>
            <execution>
                <id>prepare-agent</id>
                <goals>
                    <goal>prepare-agent</goal>
                </goals>
            </execution>
            <execution>
                <id>report</id>
                <phase>prepare-package</phase>
                <goals>
                    <goal>report</goal>
                </goals>
            </execution>
            <execution>
                <id>post-unit-test</id>
                <phase>test</phase>
                <goals>
                    <goal>report</goal>
                </goals>
                <configuration>
                    <!-- Sets the path to the file which contains the execution data. -->

                    <dataFile>target/jacoco.exec</dataFile>
                    <!-- Sets the output directory for the code coverage report. -->
                    <outputDirectory>target/jacoco-ut</outputDirectory>
                </configuration>
            </execution>
        </executions>
        <configuration>
            <systemPropertyVariables>
                <jacoco-agent.destfile>target/jacoco.exec</jacoco-agent.destfile>
            </systemPropertyVariables>
        </configuration>
    </plugin>
Solution

  • It runs successfully but the Dynamic Tests are not covered in the test coverage report generated by jacoco.

    jacoco-maven-plugin:report doesn't display coverage in tests, it displays coverage of a target of a test - in your case target seems to be class Status, whose definition is completely absent.

    Generic answer on questions like "does JaCoCo support JUnit 5 Dynamic Test?" is: JaCoCo records fact that code was executed independently from the way how it was executed - via JUnit 4 or JUnit 5, or even manually or whatever the other way of execution.

    And generic answer on questions like "why some code is not marked as covered?" is: make sure that code is actually executed, in case of automatic tests this means - make sure that these tests are actually executed.

    But let's give a try with JUnit 5 Dynamic Test. In absence of src/main/java/Status.java I'll assume that it is something like

    public class Status {

  private String code;

  public void setCode(String code) {
    this.code = code;
  }

  public String getCode() {
    return code;
  }

}

    No idea why you need systemPropertyVariables, snapshot version of jacoco-maven-plugin, multiple executions of report and redundant specification of dataFile with its default value, so will also assume that complete pom.xml after slight cleanup looks like

    <?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
  <modelVersion>4.0.0</modelVersion>

  <groupId>org.example</groupId>
  <artifactId>example</artifactId>
  <version>1.0-SNAPSHOT</version>

  <properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
  </properties>

  <dependencies>
    <dependency>
      <groupId>org.junit.jupiter</groupId>
      <artifactId>junit-jupiter-api</artifactId>
      <version>5.2.0</version>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>org.junit.jupiter</groupId>
      <artifactId>junit-jupiter-engine</artifactId>
      <version>5.2.0</version>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>junit</groupId>
      <artifactId>junit</artifactId>
      <version>4.12</version>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>org.junit.vintage</groupId>
      <artifactId>junit-vintage-engine</artifactId>
      <version>5.2.0</version>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>org.junit.platform</groupId>
      <artifactId>junit-platform-runner</artifactId>
      <version>1.2.0</version>
      <scope>test</scope>
    </dependency>
  </dependencies>

  <build>
    <plugins>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-compiler-plugin</artifactId>
        <version>3.7.0</version>
        <configuration>
          <source>1.8</source>
          <target>1.8</target>
        </configuration>
      </plugin>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-surefire-plugin</artifactId>
        <version>2.21.0</version>
      </plugin>
      <plugin>
        <groupId>org.jacoco</groupId>
        <artifactId>jacoco-maven-plugin</artifactId>
        <version>0.8.1</version>
        <configuration>
          <outputDirectory>target/jacoco-ut</outputDirectory>
        </configuration>
        <executions>
          <execution>
            <goals>
              <goal>prepare-agent</goal>
              <goal>report</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>

</project>

    After placing

    import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.DynamicTest;
import org.junit.jupiter.api.TestFactory;
import org.junit.platform.runner.JUnitPlatform;
import org.junit.platform.suite.api.SelectClasses;
import org.junit.platform.suite.api.SelectPackages;
import org.junit.runner.RunWith;

import java.util.Arrays;
import java.util.List;
import java.util.stream.Stream;

import static org.junit.Assert.assertEquals;

@RunWith(JUnitPlatform.class)
@SelectPackages("com.troll.jpt.abc.model")
@SelectClasses({Status.class})
public class DynamicModelTester {

  private Status status;

  @BeforeEach
  public void setUp() {
    status = new Status();
  }

  @TestFactory
  public Stream<DynamicTest> dynamicTestsFromStream() {
    List<String> input = Arrays.asList("abc");
    List<String> output = Arrays.asList("abc");

    status.setCode(input.get(0));

    return input.stream().map(str ->  DynamicTest.dynamicTest("status test", () -> {
      assertEquals(output.get(0), status.getCode());
    }));
  }

}

    into src/test/java/DynamicModelTester.java and execution of mvn clean verify:

    [INFO] --- maven-surefire-plugin:2.21.0:test (default-test) @ example ---
[INFO]
[INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ example ---
[INFO] Building jar: /private/tmp/jacoco/target/example-1.0-SNAPSHOT.jar
[INFO]
[INFO] --- jacoco-maven-plugin:0.8.1:report (default) @ example ---
[INFO] Skipping JaCoCo execution due to missing execution data file.

    Last line is quite suspicious as well as absence of number of tests executed by maven-surefire-plugin together with absence of target/surefire-reports/.

    Let's rename DynamicModelTester into DynamicModelTest and execute mvn clean verify again:

    [INFO] --- maven-surefire-plugin:2.21.0:test (default-test) @ example ---
[INFO]
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running DynamicModelTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.023 s - in DynamicModelTest
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO]
[INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ example ---
[INFO] Building jar: /private/tmp/jacoco/target/example-1.0-SNAPSHOT.jar
[INFO]
[INFO] --- jacoco-maven-plugin:0.8.1:report (default) @ example ---
[INFO] Loading execution data file /private/tmp/jacoco/target/jacoco.exec
[INFO] Analyzed bundle 'example' with 1 classes

    Compared to previous attempt test was executed this time. And coverage report target/jacoco-ut/default/Status.html looks like:

    coverage report

    I believe that the reason of the fact that test is not executed lies in a default value of includes of maven-surefire-plugin :

    A list of elements specifying the tests (by pattern) that should be included in testing. When not specified and when the test parameter is not specified, the default includes will be

    <includes>
    <include>**/Test*.java</include>
    <include>**/*Test.java</include>
    <include>**/*Tests.java</include>
    <include>**/*TestCase.java</include>
</includes>

    DynamicModelTester.java doesn't match any of these patters, while DynamicModelTest.java matches second, but I'll leave verification of this as a little exercise.