I manage a quite large open-source project with many unit-tests (~200 files) and passing all tests is quite time consuming for the continuous integration. We use cmake/ctest/Catch2 for the unit-test framework.
Is there a way to tell cmake/ctest to only build and run a random subset of the unit tests (e.g. just 30%) ?
When iterating with several commits on the code for a given feature, the probability that all tests where checked tends to one, but each individual commit would be way faster.
Obviously, this ratio would be set to 100% when preparing a PR or a release.
ALTETRNATIVE
Finally, I came up with a cmake solution just by creating a new add_test() function that activate a test upon a random test:
function(my_add_test test_file) #optional_avoid_add_test
string(RANDOM LENGTH 2 ALPHABET "0123456789" _random)
if (${_random} LESS ${THRESHOLD_RANDOM_TESTING})
add_executable(${test_file} ${test_file}.cpp)
add_test(${test_file} ${test_file})
endif()
endfunction()
In my main cmake I have the global variable (that can be set at cmake CLI/GUI)
SET(THRESHOLD_RANDOM_TESTING "100" CACHE INTERNAL "~% of unit tests to build and run.")
Each time I regenerate the project, a new random selection is constructed.