Search code examples
testingcmakeregressiontargetctest

Continue after dependent target fails in custom CMake target

I have a regression test suite consisting of multiple custom targets created with add_custom_target(). Moreover there is a "convenience" target regressions to run all regressions. It simply contains all single regression targets as dependency. This is reflected in the following MCVE:

cmake_minimum_required(VERSION 3.17)

project (Regressions)

add_custom_target(reg_1 COMMAND ${CMAKE_COMMAND} -E echo 'First regression')
add_custom_target(reg_2 COMMAND ${CMAKE_COMMAND} -E echo 'Second regression')
# ...
add_custom_target(regressions DEPENDS reg_1 reg_2)

Now I can run cmake --build . --target regressions from the build directory and reg_1 and reg_2 are run as part of regressions. My problem is that if one of the regressions fail, the remaining are not executed. But of course I want to always run all regressions and only have a summary of the failed ones. How can I achieve this behavior, i.e. always execute all subtargets, no matter whether some of them fail?

I assume that the natural way to do this is to use add_test() (after all regressions runs are tests), but I failed because the custom targets are no executables and AFAIK you cannot use custom CMake targets with add_test().

Please feel free to recommend an alternative to my current approach. If I could handle everything using ctest that would be preferred anyway.

Solution

  • Thanks to @KamilCuk's answer, I realized my problem of not being able to add custom targets as tests is not really a problem.

    I can invoke CMake with add_test(), and the cmake command can run custom targets.

    Adding:

    enable_testing()
add_test(NAME regression1 COMMAND ${CMAKE_COMMAND} --build ${CMAKE_BINARY_DIR} --target reg_1)
add_test(NAME regression2 COMMAND ${CMAKE_COMMAND} --build ${CMAKE_BINARY_DIR} --target reg_2)

    to the code above solves the problem. Additionally, instead of depending on the single custom targets, regression can simply invoke ctest -R "regression*" to invoke all (and only) regression targets, in case other tests exist in the CMake project, like this:

    add_custom_target(regressions COMMAND ${CMAKE_CTEST_COMMAND} -R "regression*")