gcov produces different results on Clang and GCC

Question

I'm trying to understand how to properly structure a C++ project by using CMake, googletest, and gcov for test coverage. I would like to build a general CMakeLists.txt that would work for any platform/compiler.

This is my first attempt. However, if I try to build the project and then run lcov (to generate the report), I see that I have different results if I use CLang (right result) or GCC (wrong result). Note that I'm on MacOs and I installed gcc through brew (brew install gcc).

Moreover I used the following flags in my main CMakeLists.txt:

if(CODE_COVERAGE)
    SET(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -fprofile-arcs -ftest-coverage" )
endif()

Note: If you find something wrong/weird in my CMakeLists.txt files or lcov usage, I'm open to any kind of feedback!

My library

#include "library.h"

#include <iostream>

void foo(){
    std::cout << "Foo!" << std::endl;
}

void bar(int n){
    if (n > 0){
        std::cout << "n is grater than 0!" << std::endl;
    }
    else if (n < 0){
        std::cout << "n is less than 0!" << std::endl;
    }
    else{
        std::cout << "n is exactly 0!" << std::endl;
    }
}

void baz(){  // LCOV_EXCL_START
    std::cout << "Baz!" << std::endl;
}
// LCOV_EXCL_STOP

My tests

#ifndef GCOV_TUTORIAL_TEST_LIBRARY_H
#define GCOV_TUTORIAL_TEST_LIBRARY_H

#include "../src/library.h"

#include <gtest/gtest.h>


namespace gcov_tutorial::tests {
    TEST(TestFooSuite,TestFoo){
        foo();
    }
    TEST(TestBarSuite,TestBarGreaterThanZero){
        bar(100);
    }
    TEST(TestBarSuite,TestBarEqualToZero){
        //bar(0);
    }
    TEST(TestBarSuite,TestBarLessThanZero){
        bar(-100);
    }
}

#endif //GCOV_TUTORIAL_TEST_LIBRARY_H

CLang Compilation

#!/bin/bash

# Rationale: https://vaneyckt.io/posts/safer_bash_scripts_with_set_euxo_pipefail/
set -euxo pipefail

# BASE_DIR is the project's directory, containing the src/ and tests/ folders.
BASE_DIR=$PWD
COVERAGE_FILE=coverage.info

GCOV_PATH=/usr/bin/gcov
CLANG_PATH=/usr/bin/clang
CLANGPP_PATH=/usr/bin/clang++

rm -rf build
mkdir build && cd build

# Configure
cmake -DCMAKE_C_COMPILER=$CLANG_PATH -DCMAKE_CXX_COMPILER=$CLANGPP_PATH -DCODE_COVERAGE=ON -DCMAKE_BUILD_TYPE=Release ..

# Build (for Make on Unix equivalent to `make -j $(nproc)`)
cmake --build . --config Release

# Clean-up for any previous run.
rm -f $COVERAGE_FILE
lcov --zerocounters --directory .
# Run tests
./tests/RunTests
# Create coverage report by taking into account only the files contained in src/
lcov --capture --directory tests/ -o $COVERAGE_FILE --include "$BASE_DIR/src/*" --gcov-tool $GCOV_PATH
# Create HTML report in the out/ directory
genhtml $COVERAGE_FILE --output-directory out
# Show coverage report to the terminal
lcov --list $COVERAGE_FILE
# Open HTML
open out/index.html

GCC Compilation

#!/bin/bash

# Rationale: https://vaneyckt.io/posts/safer_bash_scripts_with_set_euxo_pipefail/
set -euxo pipefail

# BASE_DIR is the project's directory, containing the src/ and tests/ folders.
BASE_DIR=$PWD
COVERAGE_FILE=coverage.info

GCOV_PATH=/usr/local/bin/gcov-11
GCC_PATH=/usr/local/bin/gcc-11
GPP_PATH=/usr/local/bin/g++-11

rm -rf build
mkdir build && cd build

# Configure
cmake -DCMAKE_C_COMPILER=$GCC_PATH -DCMAKE_CXX_COMPILER=$GPP_PATH -DCODE_COVERAGE=ON -DCMAKE_BUILD_TYPE=Release ..

# Build (for Make on Unix equivalent to `make -j $(nproc)`)
cmake --build . --config Release

# Clean-up for any previous run.
rm -f $COVERAGE_FILE
lcov --zerocounters --directory .
# Run tests
./tests/RunTests
# Create coverage report by taking into account only the files contained in src/
lcov --capture --directory tests/ -o $COVERAGE_FILE --include "$BASE_DIR/src/*" --gcov-tool $GCOV_PATH
# Create HTML report in the out/ directory
genhtml $COVERAGE_FILE --output-directory out
# Show coverage report to the terminal
lcov --list $COVERAGE_FILE
# Open HTML
open out/index.html

Answer 1

You are actually asking two questions, here.

1. Why do the coverage results differ between these two compilers?
2. How do I structure a CMake project for code coverage?

Answer 1: Coverage differences

The simple answer here is that you are building in Release mode, rather than RelWithDebInfo mode. GCC does not put as much debugging information in by default as Clang does. On my system, adding -DCMAKE_CXX_FLAGS="-g" to your build-and-run-cov-gcc.sh script yields the same results as Clang, as does building in RelWithDebInfo.

For whatever reason, it appears that Clang tracks more debug information either by default or when coverage is enabled. GCC does not have these same guardrails. The lesson to take away is this: collecting coverage information is a form of debugging; you must use a debugging-aware configuration for your compiler if you want accurate results.

Answer 2: Build system structure

It is generally a terrible idea to set CMAKE_CXX_FLAGS inside your build. That variable is intended to be a hook for your build's users to inject their own flags. As I detail in another answer on this site, the modern approach to storing such settings is in the presets

I would get rid of the if (CODE_COVERAGE) section of your top-level CMakeLists.txt and then create the following CMakePresets.json file:

{
  "version": 4,
  "cmakeMinimumRequired": {
    "major": 3,
    "minor": 23,
    "patch": 0
  },
  "configurePresets": [
    {
      "name": "gcc-coverage",
      "displayName": "Code coverage (GCC)",
      "description": "Enable code coverage on GCC-compatible compilers",
      "binaryDir": "${sourceDir}/build",
      "cacheVariables": {
        "CMAKE_BUILD_TYPE": "RelWithDebInfo",
        "CMAKE_CXX_FLAGS": "-fprofile-arcs -ftest-coverage"
      }
    }
  ],
  "buildPresets": [
    {
      "name": "gcc-coverage",
      "configurePreset": "gcc-coverage",
      "configuration": "RelWithDebInfo"
    }
  ]
}

Then your build script can be simplified considerably.

#!/bin/bash

# Rationale: https://vaneyckt.io/posts/safer_bash_scripts_with_set_euxo_pipefail/
set -euxo pipefail

# Set up defaults for CC, CXX, GCOV_PATH
export CC="${CC:-gcc-11}"
export CXX="${CXX:-g++-11}"
: "${GCOV_PATH:=gcov-11}"

# Record the base directory
BASE_DIR=$PWD

# Clean up old build
rm -rf build

# Configure
cmake --preset gcc-coverage

# Build
cmake --build --preset gcc-coverage

# Enter build directory
cd build

# Clean-up counters for any previous run.
lcov --zerocounters --directory .

# Run tests
./tests/RunTests

# Create coverage report by taking into account only the files contained in src/
lcov --capture --directory tests/ -o coverage.info --include "$BASE_DIR/src/*" --gcov-tool $GCOV_PATH

# Create HTML report in the out/ directory
genhtml coverage.info --output-directory out

# Show coverage report to the terminal
lcov --list coverage.info

# Open HTML
open out/index.html

The key here is the following lines:

# Configure
cmake --preset gcc-coverage
# Build
cmake --build --preset gcc-coverage

This script now lets you vary the compiler and coverage tool via environment variables and the CMakeLists.txt doesn't have to make any assumptions about what compiler is being used.

On my (Linux) system, I can run the following commands successfully:

$ CC=gcc-12 CXX=g++-12 GCOV=gcov-12 ./build-and-run-cov.sh

$ CC=clang-13 CXX=clang++-13 GCOV=$PWD/llvm-cov-13.sh ./build-and-run-cov.sh

Where llvm-cov-13.sh is a wrapper for llvm-cov-13 for compatibility with the --gcov-tool flag. See this answer for more detail.

#!/bin/bash
exec llvm-cov-13 gcov "$@"

As you can see, the results are indistinguishable now that the correct flags are used.