Search code examples
language-agnostictestingconventions

Should I write tests that directly map input to output, or dynamically generate a result?


It seems to me that there are two ways to write test cases:

  1. Map fixed inputs to fixed outputs and assert that they match
  2. Map generated or fixed inputs to dynamic outputs and assert that they match

# Static testing:
assert_equal "test".upcase, "TEST"

# Dynamic testing:
assert_equal person.full_name, "#{person.title} #{person.first_name} #{person.last_name}"

Of course there are pros and cons to each approach. Duplicating implementation details feels wrong, but allows me to generate sample data and run tests on it. Hard coding values makes the correct output very explicit, but doesn't seem to lend it self to reusing code.

Is the former the conventional way to write tests? Do you mix and match approaches? Is the latter method avoided for a good reason that I haven't thought of?


Solution

  • The second approach, using tools like check or quickcheck, but...

    Duplicating implementation details feels wrong

    If your duplicating then you're doing it wrong. In your code you write what the task is and how how the task is performed to a precise degree. In the test you give some invariants about the result.

    EDIT

    A note on Known Answer Tests (KATs)

    Like most generalizations, there are situations where this doesn't apply. One big area where KATs are dominant over random test vectors is cryptography (e.g. block ciphers) because there aren't supposed to be many visible invariants outside of what most type systems enforce (ex: block size). One property to check would be decrypt(key,encrypt(key,msg)) == msg.

    Simple geometry has a somewhat different problem in that no set of invariants is really a good check - you can say 0 < area(triangle) < triangle.width * triangle.height but that's just as bad. What I'm getting at here is you should be writing tests for a slightly higher level of code - something more complex that actually have a good chance of changing or being deceptively wrong.

    Situations For Random Test Vectors Some properties of code that indicate a good place for quick check properties include

    1. determinism
    2. Non-trivial
    3. clear invariants

    Trivial example using concatenation (combining two lists in series to form one new list):

    Say I have a function concat(xs,ys) = xs ++ ys. What can I check? Anything I expect to be true! Length? Yes! Elements? Yes!

    prop_len(xs,ys) = len(xs) + len(ys) = len(concat(xs,ys))
    prop_elem(xs,ys) =
        let cs = concat(xs,ys)
        elem(head xs, cs) && elem(head ys, cs) && prop_elem(tail xs,ys) && prop_elem(xs,tail ys)
    // Yes, I left out the error checking for empty list, sue me.
    

    Get the drift?