Search code examples
karate

Karate summary reports not showing all tested features after upgrade to 1.0.0

I have recently upgraded to version 1.0.0 from 0.9.6 and noticed that the generated karate-summary.html file, it doesn't display all the tested feature files in the JUnit 5 Runner unlike in 0.9.6.

What it displays instead was the last tested feature file only.

The below screenshots are from the provided SampleTest.java sample code (excluding other Tests for simplicity).

package karate;

import com.intuit.karate.junit5.Karate;

class SampleTest {

    @Karate.Test
    Karate testSample() {
        return Karate.run("sample").relativeTo(getClass());
    }
    
    @Karate.Test
    Karate testTags() {
        return Karate.run("tags").relativeTo(getClass());
    }

}

This is from Version 0.9.6. Version 0.9.6

And this one is from Version 1.0.0 Version 1.0.0

However, when running the test below in 1.0.0, all the features are displayed in the summary correctly.

@Karate.Test
Karate testAll() {
    return Karate.run().relativeTo(getClass());
}

Would anyone be kind to confirm if they are getting the similar result? It would be very much appreciated.

Version 1.0.0 testAll()

Solution

  • What it displays instead was the last tested feature file only.

    This is because for each time you run a JUnit method, the reports directory is backed up by default. Look for other directories called target/karate-reports-<timestamp> and you may find your reports there. So maybe what is happening is that you have multiple JUnit tests that are all running, so you see this behavior. You may be able to over-ride this behavior by calling the method: .backupReportDir(false) on the builder. But I think it may not still work - because the JUnit runner has changed a little bit. It is designed to run one method at a time, when you are in local / dev-mode.

    So the JUnit runner is just a convenience. You should use the Runner class / builder for CI execution, and when you want to run multiple tests and see them in one report: https://stackoverflow.com/a/65578167/143475

    Here is an example: ExamplesTest.java

    But in case there is a bug in the JUnit runner (which is quite possible) please follow the process and help the project developers replicate and then fix the issue to release as soon as possible.

    EDIT - for those looking for tips on how to "merge" reports, refer to the answers to this question: How to rerun failed features in karate?