Search code examples
compilationclangubuntu-22.04

How should I interpret the results of the `check-clang` regression tests?


I am building and running Clang for the first time, following the official "Getting Started" instructions. When I run make check-clang, it successfully runs the tests and prints the results.

However, I'm not sure how to interpret the results; do the Clang docs contain 'expected test results' that I can use to compare my results against, and if not, how do I know if Clang is 100% correctly compiled / won't cause surprises when I use it to build Python libraries? For example, should I expect 149 unsupported tests, or will that number cause problems down the road? Seems like I can assume 8 skipped tests is completely normal, but perhaps not?

Here's my test output, in case that's helpful

[100%] Running the Clang regression tests
Testing Time: 25668.06s
Total Discovered Tests: 45988
  Skipped          :     8 (0.02%)
  Unsupported      :   149 (0.32%)
  Passed           : 45801 (99.59%)
  Expectedly Failed:    30 (0.07%)

A side note, I did see some warnings during the make compilation process (While running make but prior to running make check-clang) but from what I've read on the llvm repo issues page, these errors are expected and seem to be an unresolved issue with Clang that shouldn't cause any problems down the road.

What I've tried I searched stackoverflow and the web for "expected results of clang regression tests" and similar queries, to no avail. Ditto GPT.

Thank you for any advice on this particular problem, and also for recommended methods for solving this type of problem on my own, if I've missed something obvious! 🙏

System info

Operating System: Ubuntu 22.04.5 LTS                      
          Kernel: Linux 5.15.167.4-microsoft-standard-WSL2

Solution

  • The output shown is perfectly normal, and you have successfully run clang tests. The build was successful. Here is a description of the categories:

    • Skipped/Unsupported: These tests are intentionally skipped. An example could be an end-to-end test for ARM target would be "unsupported" on an X86 machine. Similarly, a test requiring a GPU may be skipped if the GPU is absent.
    • Passed: Obviously, these tests passed.
    • Expectedly Failed: These tests are not skipped, instead they are executed, and they fail. These are known failures and it would be unexpected if they did not fail for your execution.

    The two "error" categories you are not seeing are:

    • Unexpectedly Passed: A test that is expected to fail, but passes. This is an "error". (I am not sure if this is what the category is called, but it is something similar).
    • Failed: Obviously, the test fails.

    About the warnings: I think you should see these warnings only if you are using GCC for compiling the compiler (sorry for the confusing sentence). Clang has it's own set of warnings, and LLVM project ensures that warnings are not seen (or minimal) if clang is used for the build. These are not issues with clang, but issues with the warning implementation in GCC. They are "false positives" which means that the compiler (GCC) thinks that a warning should be emitted when there should be no such warning in the first place. You can use CMAKE_C_COMPILER and CMAKE_CXX_COMPILER for setting clang as the compiler for building LLVM.