Search code examples
jenkinsjunit

How do you label flaky tests using junit?

How do I label flaky tests in junit xml syntax that jenkins uses to give us a report? jenkins gives me a nice report of tests that succeeded and failed. I would like to know which tests are known to be flaky.

Solution

  • Certainly a flaky test is something that needs to be avoided and fixed. Tests should always pass and if not they need to be worked on.
    However we don't live in a perfect world and thus it might be helpful to identify tests that have failed every now and then in the past.

    Since Jenkins keeps track of test results you can step through the history of a particular test case (using the "Previous/Next build" links).

    Additionally there are two Jenkins plugins that might be helpful:

    1. Test Results Analyzer Plugin: this plugin lets you see the history of a particular test case (or test suite) at one glance (plus adds nice charts)
    2. Flaky Test Handler Plugin: this plugin has deeper support for failing tests (i.e. rerunning them). It's a bit restricted to maven and git though.