I'm searching for clarification on the behaviour of library search paths on linux binaries when using the install directive in cmake.
For context and to simplify, we have a binary and a library that are located in the same directory.
We've been narrowing down some unusual behaviour with the program that works in some environments, and not in others. Our problems all began when we added the following line to our CMAKE file:
set(CMAKE_SKIP_RPATH TRUE)
After adding this line, the program no longer worked, and we saw this error:
binaryfile: error while loading shared libraries: mylibrary.so: cannot open shared object file: No such file or directory
We ended up narrowing the problem down to the inclusion of the install command, which I'm assuming is effectively undone by the SKIP_RPATH addition.
After a lot of mucking around to diagnose why the code worked with and without the RPATH setting, we discovered what's causing the issue, but I don't understand why this can happen, based on the rpath in the binary.
CMakeLists.txt
cmake_minimum_required (VERSION 3.15)
project (binaryfile)
set(CMAKE_RUNTIME_OUTPUT_DIRECTORY "${PROJECT_BINARY_DIR}/runtime")
link_directories (${PROJECT_BINARY_DIR}/../mylibrary)
link_libraries(mylibrary)
add_executable(binaryfile program.cpp)
# The following line allows the binary to find mylibrary in the same directory
install (TARGETS binaryfile DESTINATION bin)
After building, and moving the .so file from its build location to the same folder as the binary, this is the output of ldd and readelf
On the binary build with install ...
readelf -d binaryfile
Dynamic section at offset 0xdc0 contains 30 entries:
Tag Type Name/Value
0x0000000000000001 (NEEDED) Shared library: [libmylibrary.so]
0x0000000000000001 (NEEDED) Shared library: [libstdc++.so.6]
0x0000000000000001 (NEEDED) Shared library: [libm.so.6]
0x0000000000000001 (NEEDED) Shared library: [libgcc_s.so.1]
0x0000000000000001 (NEEDED) Shared library: [libc.so.6]
0x000000000000001d (RUNPATH) Library runpath: [/home/dev/rpath-test/program/../mylibrary:]
0x000000000000000c (INIT) 0x4004b8
0x000000000000000d (FINI) 0x400654
0x0000000000000019 (INIT_ARRAY) 0x600db0
0x000000000000001b (INIT_ARRAYSZ) 8 (bytes)
0x000000000000001a (FINI_ARRAY) 0x600db8
0x000000000000001c (FINI_ARRAYSZ) 8 (bytes)
0x0000000000000004 (HASH) 0x400298
0x000000006ffffef5 (GNU_HASH) 0x4002c8
0x0000000000000005 (STRTAB) 0x400380
0x0000000000000006 (SYMTAB) 0x4002f0
0x000000000000000a (STRSZ) 191 (bytes)
0x000000000000000b (SYMENT) 24 (bytes)
0x0000000000000015 (DEBUG) 0x0
0x0000000000000003 (PLTGOT) 0x601000
0x0000000000000002 (PLTRELSZ) 24 (bytes)
0x0000000000000014 (PLTREL) RELA
0x0000000000000017 (JMPREL) 0x4004a0
0x0000000000000007 (RELA) 0x400470
0x0000000000000008 (RELASZ) 48 (bytes)
0x0000000000000009 (RELAENT) 24 (bytes)
0x000000006ffffffe (VERNEED) 0x400450
0x000000006fffffff (VERNEEDNUM) 1
0x000000006ffffff0 (VERSYM) 0x400440
0x0000000000000000 (NULL) 0x0
ldd binaryfile
linux-vdso.so.1 (0x00007fff6099e000)
libmylibrary.so (0x00007fdff4d91000)
libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00007fdff4a07000)
libm.so.6 => /lib64/libm.so.6 (0x00007fdff46cf000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fdff44b7000)
libc.so.6 => /lib64/libc.so.6 (0x00007fdff40fd000)
/lib64/ld-linux-x86-64.so.2 (0x00007fdff4f93000)
On the binary build without install ...
readelf -d binaryfile
Dynamic section at offset 0xdc0 contains 30 entries:
Tag Type Name/Value
0x0000000000000001 (NEEDED) Shared library: [libmylibrary.so]
0x0000000000000001 (NEEDED) Shared library: [libstdc++.so.6]
0x0000000000000001 (NEEDED) Shared library: [libm.so.6]
0x0000000000000001 (NEEDED) Shared library: [libgcc_s.so.1]
0x0000000000000001 (NEEDED) Shared library: [libc.so.6]
0x000000000000001d (RUNPATH) Library runpath: [/home/dev/rpath-test/program/../mylibrary]
0x000000000000000c (INIT) 0x4004b8
0x000000000000000d (FINI) 0x400654
0x0000000000000019 (INIT_ARRAY) 0x600db0
0x000000000000001b (INIT_ARRAYSZ) 8 (bytes)
0x000000000000001a (FINI_ARRAY) 0x600db8
0x000000000000001c (FINI_ARRAYSZ) 8 (bytes)
0x0000000000000004 (HASH) 0x400298
0x000000006ffffef5 (GNU_HASH) 0x4002c8
0x0000000000000005 (STRTAB) 0x400380
0x0000000000000006 (SYMTAB) 0x4002f0
0x000000000000000a (STRSZ) 190 (bytes)
0x000000000000000b (SYMENT) 24 (bytes)
0x0000000000000015 (DEBUG) 0x0
0x0000000000000003 (PLTGOT) 0x601000
0x0000000000000002 (PLTRELSZ) 24 (bytes)
0x0000000000000014 (PLTREL) RELA
0x0000000000000017 (JMPREL) 0x4004a0
0x0000000000000007 (RELA) 0x400470
0x0000000000000008 (RELASZ) 48 (bytes)
0x0000000000000009 (RELAENT) 24 (bytes)
0x000000006ffffffe (VERNEED) 0x400450
0x000000006fffffff (VERNEEDNUM) 1
0x000000006ffffff0 (VERSYM) 0x40043e
0x0000000000000000 (NULL) 0x0
ldd binaryfile
linux-vdso.so.1 (0x00007ffecd3b5000)
libmylibrary.so => not found
libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00007f7d9179f000)
libm.so.6 => /lib64/libm.so.6 (0x00007f7d91467000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f7d9124f000)
libc.so.6 => /lib64/libc.so.6 (0x00007f7d90e95000)
/lib64/ld-linux-x86-64.so.2 (0x00007f7d91b29000)
If mylibrary.so exists in /home/dev/rpath-test/mylibrary then both binaries work, as expected, and also report that libmylibrary.so is located in that directory. What I'm trying to understand is what is the key that's allowing the 'install' version to find the library in the pwd.
readelf shows that there is a colon at the end of the runpath. I'm guessing that this is significant - does it mean that an empty string is part of the runpath, and that it infers pwd?
Look forward to any insights into this.
I'm guessing that this is significant
Yes.
does it mean that an empty string is part of the runpath
Yes.
and that it infers pwd?
Kind of: ld-linux will path-join each :-separated component of RPATH with the name of the library, and will try to open the resulting path.
For the /home/dev/rpath-test/program/../mylibrary: RPATH, ld-linux will try to open /home/dev/rpath-test/program/../mylibrary/libmylibrary.so, and if that fails libmylibrary.so.
If libmylibrary.so exists in the current working directory, then ld-linux will find it.