Selenium - abort the tests gracefully if a catastrophic error occurs - but generate test report

Question

I have reviewed existing questions/answers about the this topic, and have not seen any that relate to my question.

To start off, my test automation uses two listeners - TestListener and CustomReportListener - along with TestNG framework with Selenium Webdriver to test a website.

Question is: How do I terminate rest of the tests (i.e. the rest of the test suite) if a critical condition is detected in one of the tests - but still generate test report that is mailed to the testers?

The test that ran into the critical issue would be marked Failed but the rest (ideally) would be marked Skipped.

The easiest way to terminate the tests is to use System.exit(1). I have tried that but it kills the test process - thereby, not creating a test report.

Answer 1

You can skip the remaining tests using your TestListener. In you test methods, if the critical condition has occurred, then you have to set an attribute in the ITestContext object:

@Test
public void testCase(ITestContext ctx) {
    //.....rest of the code....

    if(/*the critical condition*/) {
        ctx.setAttribute("criticalFailure", true);
    }

    // or if you are expecting an exception to identify the critical issue..
    try {
        // code which you expect may throw the critical exception
    } catch (YourExpectedException e) {
        ctx.setAttribute("criticalFailure", true);
    }
}

Now in your listener, check if this attribute was set.

public class TestListener implements ITestListener {

    @Override
    public void onTestStart(ITestResult result) {
        // Note that you should not use result.getAttribute because
        // in the test method, we have set the attribute in the test context.
        // (ITestResult is not available in test methods)

        Object isCritical = result.getTestContext().getAttribute("criticalFailure");
        if (isCritical != null && (boolean) isCritical) {
            throw new SkipException("Critical error occurred");
        }
    }
}

---

If you are considering all failures as critical error, then it is much more easy. You don't have to change anything in your test methods. The only change would be in your test listener.

public class TestListener implements ITestListener {

    @Override
    public void onTestFailure(ITestResult result) {
        result.setAttribute("failed", true);
    }

    @Override
    public void onTestStart(ITestResult result) {
        Object failed = result.getAttribute("failed");
        if (failed != null && (boolean) failed) {
            throw new SkipException("Critical error occurred");
        }
    }
}