CppUnit tests always passing with Ctest

Question

I'm working on a project using CMake and just integrated some CppUnit tests. I would like to use CTest and thus I used add_test in my CMakeLists.txt files to have the tests executed when typing make test. Yet I observe that, when typing make test, it says that all the tests passed even if I make a test with trivial errors. Erroneous tests report these errors when executed manually (e.g. ./my_test) but not when executed using make test.

Here is the content of my CMakeLists.txt in the test directory:

add_executable(TestDataSpace TestDataSpace.cpp)
target_link_libraries(TestDataSpace ${DEP_LIBRARIES} ${CPPUNIT_LIBRARIES})

add_executable(TestVariableManager TestVariableManager.cpp)
target_link_libraries(TestVariableManager ${DEP_LIBRARIES} ${CPPUNIT_LIBRARIES})

add_executable(TestLayoutManager TestLayoutManager.cpp)
target_link_libraries(TestLayoutManager ${DEP_LIBRARIES} ${CPPUNIT_LIBRARIES})

add_test(NAME "TestDataSpace" COMMAND ${MY_PROJECT_SOURCE_DIR}/test/TestDataSpace)
add_test(NAME "TestVariableManager" COMMAND ${MY_PROJECT_SOURCE_DIR}/test/TestVariableManager)
add_test(NAME "TestLayoutManager" COMMAND ${MY_PROJECT_SOURCE_DIR}/test/TestLayoutManager)

CTest does find the executables, since putting a wrong path for the command makes CMake complain that it doesn't find them.

make test outputs the following:

Running tests... Test project

Start 1: TestDataSpace 1/3 Test #1: TestDataSpace ....................   Passed    0.01 sec
Start 2: TestVariableManager 2/3 Test #2: TestVariableManager ..............   Passed    0.02 sec
Start 3: TestLayoutManager 3/3 Test #3: TestLayoutManager ................   Passed    0.01 sec

100% tests passed, 0 tests failed out of 3

What am I missing?

Answer 1

I'm not familiar with CppUnit, but I suspect your executables are always returning 0, even if the test fails. CTest takes a return of 0 to indicate success.

If you change your return value when the test fails to a non-zero number, you should see the expected output from CTest.

Alternatively, you can modify CTest's behaviour by using set_tests_properties to set the values of PASS_REGULAR_EXPRESSION and/or FAIL_REGULAR_EXPRESSION. If either of these are set, the return value is ignored. So for example, you could do:

set_tests_properties(
    TestDataSpace
    TestVariableManager
    TestLayoutManager
        PROPERTIES PASS_REGULAR_EXPRESSION "TEST PASSED;Pass")

As an aside, you can avoid passing the full path to the test executables in your case since they are actual CMake targets defined in the same CMakeLists.txt:

add_test(NAME TestDataSpace COMMAND TestDataSpace)
add_test(NAME TestVariableManager COMMAND TestVariableManager)
add_test(NAME TestLayoutManager COMMAND TestLayoutManager)