I am new to the SO. I have a simple unit test code where I am doing following operations :
mysqrt library .When I am running the code with CTEST_PARALLEL_LEVEL = 1 my all test cases are passing.
But when I do CTEST_PARALLEL_LEVEL = 8 then my test cases are failing some time for some input which is not fixed in every run.
99% the ALL results are passing but 1% its failing.
Error:
mysqrt.o: file not recognized: File truncated
I have deleted the object file explicitly by using rm *.o ,But still this error is coming after few runs .
I am not sure why this error is coming with CTEST_PARALLEL_LEVEL = 8
I am attaching my CMakeList only as some of Stack Overflow expert can understand the issue by checking these 3 CMakeLists.txt files.
NOTE: As per the Stack overflow guidelines I am not attaching my source code of sqrt and addition function to avoid the bigger length of the question .
My folder structure:
SAMPLE_TEST
├── CMakeLists.txt
├── MathFunctions
│ ├── CMakeLists.txt
│ ├── MathFunctions.h
│ └── mysqrt.cpp
└── unit_test
├── CMakeLists.txt
└── step2
├── CMakeLists.txt
├── execute.cpp
└── tutorial.cpp
SAMPLE_TEST
CMakeLists.txt
cmake_minimum_required(VERSION 3.1)
project(Tutorial)
ENABLE_TESTING()
add_subdirectory(MathFunctions)
add_subdirectory(unit_test)
MathFunctions folder
CMakeLists.txt
add_library(MathFunctions mysqrt.cpp)
set(REF_FILES mysqrt.cpp)
add_definitions(-Wall -Wextra -pedantic -std=c++11)
add_custom_target(build_reference_library
DEPENDS sqrtlib
COMMENT "Generating sqrtlib")
ADD_LIBRARY(sqrtlib OBJECT ${REF_FILES})
unit_test folder
CMakeLists.txt
set(REF_MATHLIB_DIR ${CMAKE_CURRENT_SOURCE_DIR}/../MathFunctions)
macro(GENERATION file input)
set(ip_generator ctest_input_${input})
add_executable(${ip_generator}
${file}
$<TARGET_OBJECTS:sqrtlib>
)
target_compile_options(${ip_generator} PUBLIC
-Wall -Wextra -g -std=c++11
-DCTEST_INPUT=${input})
target_link_libraries(${ip_generator} PUBLIC
dl pthread
)
target_include_directories(${ip_generator} PUBLIC
${REF_MATHLIB_DIR}
)
set(INPUT_FILE0 ip0_${input}.y)
set(INPUT_FILE0_TXT ip0_${input}.txt)
add_custom_command(
OUTPUT ${INPUT_FILE0} ${INPUT_FILE0_TXT}
COMMAND ${ip_generator} > ${INPUT_FILE0_TXT}
MAIN_DEPENDENCY ${sqrtlib}
COMMENT "Generating output files of for testcase")
add_custom_target(gen_input_${input}
DEPENDS ${INPUT_FILE0}
COMMENT "Generated output files")
endmacro()
####################
macro(EXECUTE file input)
get_filename_component(main_base_name ${file} NAME_WE)
set(main_base_name_mangled ${main_base_name}_${input})
set(exe_generator ctest_ref_${input})
add_executable(${exe_generator}
${file}
$<TARGET_OBJECTS:sqrtlib>
)
target_compile_options(${exe_generator} PUBLIC
-Wall -Wextra -g -std=c++11
-DCTEST_INPUT=${input})
target_link_libraries(${exe_generator} PUBLIC
dl pthread
)
target_include_directories(${exe_generator} PUBLIC
${REF_MATHLIB_DIR}
)
set(INPUT_FILE0 ip0_${input}.y)
set(EXE_FILE0 exeadd_${input}.y)
set(EXE_FILE_TXT exeadd_${input}.txt)
add_custom_command(
OUTPUT ${EXE_FILE0} ${EXE_FILE_TXT}
COMMAND ${exe_generator} > ${EXE_FILE_TXT}
MAIN_DEPENDENCY ${INPUT_FILE0} ${sqrtlib}
COMMENT "Generating output files of for testcase")
add_custom_target(gen_execute_${input}
DEPENDS ${EXE_FILE0}
COMMENT "Generated output files")
# add test to simulate
add_test(NAME ctest_execute_${input}
COMMAND ${CMAKE_COMMAND} --build ${CMAKE_BINARY_DIR}
--target gen_execute_${input})
#add_dependencies(execute_${main_base_name_mangled}
#gen_input)
endmacro()
#++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++#
# add test directories
#++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++#
set(TEST_DIRECTORIES
step2
)
foreach(dir ${TEST_DIRECTORIES})
add_subdirectory(${dir})
endforeach()
step2 folder
CMakeLists.txt
set(UT_IPGEN_FILES tutorial.cpp)
set(UT_EXECUTE_FILES execute.cpp)
set(input_integer_range 1 4 9 16 25 36 49 64 81 100 121 144 )
foreach(ip_integer ${input_integer_range})
GENERATION(${UT_IPGEN_FILES} ${ip_integer})
EXECUTE(${UT_EXECUTE_FILES} ${ip_integer})
endforeach(ip_integer)
Result: 1st Run:
Start 1: ctest_execute_1
Start 2: ctest_execute_4
Start 3: ctest_execute_9
Start 4: ctest_execute_16
Start 5: ctest_execute_25
Start 6: ctest_execute_36
Start 7: ctest_execute_49
Start 8: ctest_execute_64
1/12 Test #4: ctest_execute_16 .................***Failed 1.14 sec
2/12 Test #6: ctest_execute_36 ................. Passed 1.27 sec
3/12 Test #7: ctest_execute_49 ................. Passed 1.32 sec
4/12 Test #8: ctest_execute_64 ................. Passed 1.32 sec
Start 9: ctest_execute_81
Start 10: ctest_execute_100
Start 11: ctest_execute_121
Start 12: ctest_execute_144
5/12 Test #1: ctest_execute_1 .................. Passed 1.33 sec
6/12 Test #2: ctest_execute_4 .................. Passed 1.33 sec
7/12 Test #3: ctest_execute_9 .................. Passed 1.33 sec
8/12 Test #5: ctest_execute_25 ................. Passed 1.33 sec
9/12 Test #10: ctest_execute_100 ................ Passed 0.54 sec
10/12 Test #11: ctest_execute_121 ................ Passed 0.55 sec
11/12 Test #9: ctest_execute_81 ................. Passed 0.55 sec
12/12 Test #12: ctest_execute_144 ................ Passed 0.55 sec
92% tests passed, 1 tests failed out of 12
Total Test time (real) = 1.88 sec
The following tests FAILED:
4 - ctest_execute_16 (Failed)
2nd Run:
Start 1: ctest_execute_1
Start 2: ctest_execute_4
Start 3: ctest_execute_9
Start 4: ctest_execute_16
Start 5: ctest_execute_25
Start 6: ctest_execute_36
Start 7: ctest_execute_49
Start 8: ctest_execute_64
1/12 Test #6: ctest_execute_36 ................. Passed 1.31 sec
2/12 Test #7: ctest_execute_49 ................. Passed 1.36 sec
3/12 Test #8: ctest_execute_64 ................. Passed 1.36 sec
Start 9: ctest_execute_81
Start 10: ctest_execute_100
Start 11: ctest_execute_121
4/12 Test #1: ctest_execute_1 .................. Passed 1.37 sec
5/12 Test #2: ctest_execute_4 .................. Passed 1.37 sec
6/12 Test #3: ctest_execute_9 .................. Passed 1.36 sec
7/12 Test #4: ctest_execute_16 ................. Passed 1.36 sec
8/12 Test #5: ctest_execute_25 ................. Passed 1.37 sec
Start 12: ctest_execute_144
9/12 Test #11: ctest_execute_121 ................ Passed 0.50 sec
10/12 Test #10: ctest_execute_100 ................ Passed 0.51 sec
11/12 Test #9: ctest_execute_81 ................. Passed 0.51 sec
12/12 Test #12: ctest_execute_144 ................ Passed 0.34 sec
100% tests passed, 0 tests failed out of 12
Total Test time (real) = 2.01 sec
Your tests executing
COMMAND ${CMAKE_COMMAND} --build ${CMAKE_BINARY_DIR} --target ...
which effectively runs make (or whatever build tool you use) in the project's build directory.
But concurrent invocations of make in the same directory are never guarantee to work correctly. This is why you got weird errors when run tests in parallel (with CTEST_PARALLEL_LEVEL variable being set).
E.g. all these tests are trying to create the same object file mysqrt.o, and this creation is definitely not thread-safe.
By running
make sqrtlib
before
ctest
you may be sure that the object file is already created when tests are run, and tests wouldn't attempt to create it again. But you still could get other conflicts in the parallel tests.
It depends on what actually you want to check by the testing, but usually a test checks behavior of some program or library, and it doesn't intend to check a compilation(building) of that program. Because of that, compilation(building) commands are performed before the testing.
Usually it is convenient to follow(implement) this workflow for testing:
# Configure the project
cmake <source-directory>
# Build the project.
# It builds both program/library intended, and the tests themselves.
make
# run tests
ctest <params>
In that case a test could have the following definition:
add_test(NAME ctest_execute_${input} COMMAND ${exe_generator})
(Unless you want to check output of the test by some automatic way, no need to explicitely save this output by redirecting into the file. ctest by itself would collect the output of the test, so you may read it if needed).