Surefire only reports first test suite from module to RunListener

Question

So I have a TeamCity and JUnit tests, and I want to write a RunListener to notify TeamCity in real-time about running tests - TeamCity only supports batched test reporting for JUnit out of box.

I have several suites, annotated with @RunWith(Suite.class) for logical grouping of tests. TeamCity shows tests grouped by suite correctly. However, my problem is that Surefire will only call RunListener.testRunStarted once, with both suites' names in description (but no way to attribute tests to either one).

So I have implemented

public class JUnitTeamcityReporter extends RunListener {
    /** */
    private volatile Description suite;

    /** */
    @Override public void testRunStarted(Description desc) {
        this.suite = desc;
    }

    /** */
    @Override public void testStarted(Description desc) {
        System.out.println(String.format("##teamcity[testStarted name='%s' captureStandardOutput='true']",
            testName(suite, desc)));
    }
...

And I have hooked it up in my pom.xml:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <version>3.0.0-M3</version>
    <configuration>
        <forkCount>0</forkCount>
        <properties>
            <property>
                <name>listener</name>
                <value>com.pany.JUnitTeamcityReporter</value>
            </property>
        </properties>
    </configuration>
</plugin>

I run maven with -Dtest=FirstTestSuite,SecondTestSuite

and the output is the following:

[INFO] Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.803 s - in com.pany.FirstTestSuite
# I expect that my RunListener will be notified HERE, but it does not happen!
[INFO] Running com.pany.SecondTestSuite

Otherwise, my solution works fine if there's just one suite per module.

Is it possible to make Surefire report every Suite properly to testRunStarted? Unfortunately, there does not seem to be a way to get current running suite from test discription, so I don't understand how to work around this.

Answer 1

This information is not available to JUnit, and Surefire has rather poor extension capabilities, but it can be reached:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <version>3.0.0-M4</version> <!-- upgrade to M4 -->
    <configuration>
        <workingDirectory>${user.dir}</workingDirectory>
        <!-- surefire integration to intercept test suite start: -->
        <statelessTestsetInfoReporter 
            implementation="com.pany.TestSuiteAwareTestsetReporter"/>
        <properties>
            <property>
                <name>listener</name>
                <value>com.pany.JUnitTeamcityReporter</value>
            </property>
        </properties>
    </configuration>
    <dependencies>
    <!-- Add dependency to plugin (for surefire) as well as
    compile,test dependency to module (for junit) since they're on different classpaths -->
        <dependency>
            <groupId>com.pany</groupId>
            <artifactId>dev-tools</artifactId>
            <version>${project.version}</version>
        </dependency>
    </dependencies>
</plugin>

and the code of test suite reporter:

public class TestSuiteAwareTestsetReporter extends SurefireStatelessTestsetInfoReporter {
    @Override public StatelessTestsetInfoConsoleReportEventListener<WrappedReportEntry, TestSetStats> createListener(
        ConsoleLogger log) {
        return new ConsoleReporter(log, false, false) {
            public void testSetStarting(TestSetReportEntry report) {
                MessageBuilder builder = MessageUtils.buffer();

                /** @see TestSetStats#concatenateWithTestGroup(MessageBuilder, ReportEntry, boolean) */
                JUnitTeamcityReporter.suite = concatenateWithTestGroup(builder, report);

                super.testSetStarting(report);
            }
        };
    }

...