analysis of code coverage

Question

I am working with asp.net web forms project developed in framework 4.0. It does not have any unit tests at present.

I am currently adding a new feature to the site in which I hope to introduce unit and integration tests for. Not being familiar with the concept of code coverage I hope to achieve a high percentage of such for the new feature.

I will be using the integrated test environment supplied with visual studio and rhino mocks for mock generation. For a new feature, what level of code coverage should I look to achieving? And for a novice to code coverage, how can this be measured for a specific feature of the site?

Answer 1

It is possible to consistently get code coverage approaching 95% in unit tests, but you need to always pay attention to what you're actually testing and what you're reducing to a mockery.

Having 100% code coverage by unit tests with a flawed mock is worse than having no test coverage at all. A bad mock leads to a false sense of security - you think you know something about how your code will fare in the real world when in fact you know nothing. Because you have confidence in this false knowledge, you know less than nothing. If you had no testing at all at least then you would have fear as your guide.

Mocks are a great testing tool but you have to keep in mind that your mocks will never be more than an emulation of what you think the real world is. You generally don't find problems in the main flow of real world code. It's in the backwater eddies that the real world departs from rational expectation and where mocks will lead you into a false sense of security.

Integration testing has a better grip on reality since you're generally running against real systems. But it's extremely difficult to simulate failure cases in integration testing to sweep out all the corner cases in your code paths and squeeze out that last 5% of code coverage. Unit testing excels at forcing error conditions and exercising your code's error handling.

Be careful not to read too much into code coverage statistics. You know what they say: lies, damned lies, and code coverage statistics. ;> Code coverage data is a valuable tool for gaining insight into what your unit tests are doing, but it is not an absolute truth. Code coverage results can be superficially flashy but need to be tempered with understanding.

Few code coverage tools go to the trouble of telling you the whole story. Most simply tell you that a line of code executed at some time during the test run. They don't tell you if that line executed on all possible code paths through a chunk of code. Part of the reason is because the number of code path permutations grows exponentially with each additional if statement. That degree of growth is hard to manage, and it's hard to visualize.

Consider this code:

public class Class1
{
    public static void foo(bool a, bool b)
    {
        int x = 100;
        if (a)
            Console.WriteLine("statement a");
        else
            x -= 50;

        if (b)
            Console.WriteLine("statement c");
        else
            x -= 50;

        double y = 10 / x;
    }
}

[TestClass]
public class UnitTest1
{
    [TestMethod]
    public void TestMethod1()
    {
        Class1.foo(true, true);
        Class1.foo(true, false);
        Class1.foo(false, true);
    }
}

Code coverage will tell you that all the lines in this code snippet have been exercised by your unit test.

Congratulations! You have achieved the 100% code coverage badge of honor. Oh, and by the way, your code still contains a crash bug.

The coverage results are correct in that the unit tests did exercise every line of code in the snippet. But that is not the complete story, the complete truth. The code path of (a,b) = (false, false) passing through both else clauses in the same execution context was not executed by the unit tests. And that's the one path that will cause a division by zero error at runtime in the real world.

Don't get me wrong - unit testing is vital to sanity checking your work before letting it out to play in the shark tank. Code coverage analysis is a very powerful tool to quickly determine what areas of your code have never ever seen the light of day and might deserve a little more of your attention before release.

Use these tools and consider their results thoughtfully and they will serve you well.