Search code examples
unit-testinglanguage-agnostic

Is it possible to write a unit test that cover everything?

Let's say I have a function

function (int x) {
  if (x < 10) return true;
  return false;
}

Ideally, you want to write 2^32 - 1 test cases to cover from INT_MIN to INT_MAX? Of course this is not practical.

To make life easier, we write test cases for

  • x < 10, test x = 9 expect true
  • x == 10, test x = 10 expect false
  • x > 10, test x = 11 expect false

These test cases are fine but it does not cover every case. Let's say one day someone modified the function to be 

function (int x) {
  if (x == 12) return true;
  if (x < 10) return true;
  return false;
}

he will run the test and realize all the test passed. How do we make sure we cover every senario without going to extreme. Is there a key word for this issue I am describing?

Solution

  • This is partly a comment partly an answer because of the way you phrased the question.

    The comment

    Is it possible to write a unit test that cover everything?

    No. Even in your example you limit the test cases to 2^32 but what if the code is moved to a 64 bit system and then someone adds a line using 2^34 or something.

    Also your question indicates to me that you are thinking of static test cases with dynamic code, e.g. the code is dynamic in that it is changed over time by a programmer, this does not mean dynamically modified by the code. You should be thinking dynamic test cases with dynamic code.

    Lastly you did not note if it was white, gray or black box testing.

    The answer

    Let a tool analyze the code and generate the tests data.

    See: A Survey on Automatic Test Data Generation

    Also you asked about key words for searching.

    Here is a Google search for this that I found of value:

    code analysis automated test generation survey

    Related

    I have never used one of these test case tools myself as I use Prolog DCG to generate my test cases and currently with a project I am doing generate millions of test cases in about two minutes and test them over a few minutes. Some of the test cases that fail I would never have thought up on my own so this may be considered overkill by some, but it works.

    Since many people don't know Prolog DCGs here is a similar way explained using C# with LINQ by Eric Lippert, Every Binary Tree There Is