Search code examples
haskellcabal

Cabal says tests pass but in fact they fail?

When I run the test suite using cabal test, I got the following message:

Running 1 test suites...
Test suite tests: RUNNING...
Test suite tests: PASS
Test suite logged to: my-lib-tests.log

But when I looked at the log file, the content was:

Test suite tests: RUNNING...
*** Failed! Falsified (after 1 test):
[]
Test suite tests: PASS
Test suite logged to: my-lib-tests.log

Why did I get a pass message when the tests clearly failed?

Solution

  • cabal test works under the assumption that a failing test suite will exit with a non-zero error code.

    quickCheck prints a counterexample but returns normally.

    To make the test executable fail when a counterexample is found, you can wrap QuickCheck tests using quickCheckResult and isSuccess.

    There are test frameworks that do this for you, with a lot of useful functionality on top (like command-line arguments to select the tests to run), like tasty, with tasty-quickcheck.