Can I automatically execute JUnit testcases once with all logging enabled and once with all logging disabled?

Question

I've found a solution, see my own answer below. Does anyone have a more elegant one?

I want to do this to increase code-coverage and find subtle bugs.

Assume the following class to be tested:

public class Foo {
    private final Logger logger = LoggerFactory.getLogger(Foo.class);
    public void bar() {
        String param=[..];
        if(logger.isInfoEnabled()) logger.info("A message with parameter {}", param);

        if(logger.isDebugEnabled()) {
            // some complicated preparation for the debug message
            logger.debug([the debug message]);
        }
    }
}

and the following test-class:

public class FooTest {
    @Test
    public void bar() {
        Foo foo=new Foo();
        foo.bar();
    }
}

A code-coverage tool like e.g. Cobertura will correctly report that only some of the conditional branches have been checked.

info and debug are either activated or deactivated for the logger.

Besides looking bad in your coverage score, this poses a real risk.

What if there is some side effect caused by code inside if(logger.isDebugEnabled())? What if your code does only work if DEBUG is enabled and fails miserably if the log level is set to INFO? (This actually happened in one of our projects :p)

So my conclusion is that code containing logger statements should always be tested once with all logging enabled and once with all logging disabled...

Is there a way to do something like that with JUnit? I know how to globally enable or disable all my logging in Logback so the problem is: How can I execute the tests twice, once with logging enabled, once with logging disabled.

p.s. I'm aware of this question but I don't think this is a duplicate. I'm less concerned about the absolute coverage values but about subtle, hard-to-find bugs that might be contained inside of a if(logger.isDebugEnabled()).

Answer 1

I've solved this problem by implementing a base class that test classes should extend if such functionality is desired.

The article Writing a parameterized JUnit test contained the solution.

See LoggingTestBase for the logging base class and LoggingTestBaseExampleTest for a simple example that's using it.

Every contained test method is executed three times:

1. It's executed using the logging as defined in logback-test.xml as usual. This is supposed to help while writing/debugging the tests.

2. It's executed with all logging enabled and written to a file. This file is deleted after the test.

3. It's executed with all logging disabled.

Yes, LoggingTestBase needs documentation ;)