Why does changing the linking order fix some linking errors on one system?

So I had this strange behavior with GitLab CI. I got it working but now I am wondering why it works.

First of all I was starting with GitLab CI. I got a local runner with docker on my machine (Arch Linux), so that I can test without pushing and waiting. I wrote a test with the googletest framework (Just an assert true). I triggered the script locally and everything worked. All tests passed in the local docker image.

So now, when everything was working, I pushed to the repository and a runner took the job. This ran on Ubuntu 16.04. Now it compiled and after the execution it was throwing a segmentation fault.

I got to debugging at the Ubuntu system and after a while I switched the linking order of two libraries:

From:

target_link_libraries(${PROJECT_NAME}_test PRIVATE Threads::Threads -lglog -lpthread
    ${QT_LIBRARIES}
    ${PROJECT_BINARY_DIR}/googletest-build/googlemock/libgmock.a
    ${PROJECT_BINARY_DIR}/googletest-build/googlemock/gtest/libgtest.a
    ${OpenCV_LIBRARIES}
)

To:

target_link_libraries(${PROJECT_NAME}_test PRIVATE Threads::Threads -lglog -lpthread
    ${QT_LIBRARIES}
    ${OpenCV_LIBRARIES}
    ${PROJECT_BINARY_DIR}/googletest-build/googlemock/libgmock.a
    ${PROJECT_BINARY_DIR}/googletest-build/googlemock/gtest/libgtest.a
)

I am using CMake for the build.

Both PCs are running the same version of docker (17.12.0-ce).

The used gcc docker image is: sha256:95d81930694ca9e705b71dc141ddd13f466f4989857f74aebaf1d29ba6553775

Appareantly this question is linked: Why does the order in which libraries are linked sometimes cause errors in GCC?

Now my question: When both systems run a docker container. Why does changing the linking order in this case fix the problem?

Solution

The proper solution to this in CMake is not to adjust the order manually, but rather to model the inter-dependencies between different targets correctly.

The exact nature of the order dependency here is toolchain dependent (on gcc, dependees have to come before dependencies on the linker command line; MSVC doesn't care; other toolchains may choose different order requirements). The only way for CMake to ensure that it generates the correct order for the given toolchain is by modelling the dependencies explicitly in CMake.

In your example you have modeled a flat list of dependencies:

target_link_libraries(${PROJECT_NAME}_test PRIVATE Threads::Threads -lglog -lpthread
    ${QT_LIBRARIES}
    ${OpenCV_LIBRARIES}
    ${PROJECT_BINARY_DIR}/googletest-build/googlemock/libgmock.a
    ${PROJECT_BINARY_DIR}/googletest-build/googlemock/gtest/libgtest.a
)

You have a single target ${PROJECT_NAME}_test that depends on a bunch of libraries. But that is actually wrong! In reality, there is a dependency from gmock to gtest which you didn't tell CMake about. You need to model this dependency explicitly in order for CMake to work correctly. Since dependencies can only be specified between targets, we need to introduce two additional targets for gtest and gmock:

add_library(gtest INTERFACE)
target_link_libraries(gtest INTERFACE ${PROJECT_BINARY_DIR}/googletest-build/googlemock/gtest/libgtest.a)
add_library(gmock INTERFACE)
target_link_libraries(gmock INTERFACE ${PROJECT_BINARY_DIR}/googletest-build/googlemock/libgmock.a)
target_link_libraries(gmock INTERFACE gtest)   # now gmock depends on gtest

target_link_libraries(${PROJECT_NAME}_test PRIVATE Threads::Threads -lglog -lpthread
    ${QT_LIBRARIES}
    ${OpenCV_LIBRARIES}
    gtest
    gmock     # order doesn't matter here;
              # you can even omit gtest completely now
)

Note the target_link_libraries call here that establishes the dependency from gmock to gtest. It is very important to always model direct dependencies between static libraries like this in CMake, otherwise you will get problems like the one you described, which can quickly grow over your head once your build exceeds a certain complexity.

As a side note, try not to hard-code library paths in your CMake files, as that again makes your build non-portable and may break completely on different toolchains.