Search code examples
gcclinkerclangld

Priority of LIBRARY_PATH with custom gcc installations

Is there a way to get compilers to prefer libraries from LIBRARY_PATH instead of system paths. I am particularly looking for Clang. I partially solved the problem for GCC while writing this question, but it's also not quite clear.

Background

LIBRARY_PATH is a convenient environment variable to allow transparently linking libraries in non-standard directories, e.g. user installations, and in my case environment modules that provide different versions of a library. The idea is to do a module load libfoo/version and the compiler will transparently use the right libfoo.so.

For a shared library, one also needs to set LD_LIBRARY_PATH for ld.so to find the right library. If there are multiple libfoo.so in both LD_LIBRARY_PATH and /usr/lib, ld.so specifies that LD_LIBRARY_PATH is searched before default paths.

Problem

I have encountered issues when the library defines a soname - which is different between the two libfoo.so versions (which are symlinks to libfoo.so.1 and libfoo.so.2 respectively) in /usr/lib and LIBRARY_PATH. Then ld will link against the soname in /usr/lib and LD_LIBRARY_PATH can no longer prioritize the intended library.

I first encountered this with boost, but here's a small example:

echo "void foo() {}" > foo.c

# create an old libfoo version in /usr/lib
sudo gcc foo.c -fpic -shared -o /usr/lib/libfoo.so.1 -Wl,-soname,libfoo.so.1
sudo ln -s libfoo.so.1 /usr/lib/libfoo.so

# create the new libfoo that we want to transparently override
mkdir -p /tmp/XXX/lib
gcc foo.c -fpic -shared -o /tmp/XXX/lib/libfoo.so.2 -Wl,-soname,libfoo.so.2
ln -s libfoo.so.2 /tmp/XXX/lib/libfoo.so

export LIBRARY_PATH=/tmp/XXX/lib
export LD_LIBRARY_PATH=/tmp/XXX/lib

echo "void foo(); int main() { foo(); }" > main.c
gcc main.c -lfoo
ldd a.out| grep foo
    # under some conditions this shows libfoo.so.1 instead of .2
    libfoo.so.1 => /usr/lib/libfoo.so.1

GCC

I initially encountered this issue with a custom installation of GCC while it was working as expected for the system installation. gcc --print-search-dirs reveals a pattern:

/tmp/XXX/lib/x86_64-pc-linux-gnu/7.2.0/
/tmp/XXX/lib/x86_64-linux-gnu/
/tmp/XXX/lib/../lib64/
/opt/gcc/7.2.0/lib/gcc/x86_64-pc-linux-gnu/7.2.0/
/opt/gcc/7.2.0/lib/gcc/x86_64-pc-linux-gnu/7.2.0/../../../../x86_64-pc-linux-gnu/lib/x86_64-pc-linux-gnu/7.2.0/
/opt/gcc/7.2.0/lib/gcc/x86_64-pc-linux-gnu/7.2.0/../../../../x86_64-pc-linux-gnu/lib/x86_64-linux-gnu/
/opt/gcc/7.2.0/lib/gcc/x86_64-pc-linux-gnu/7.2.0/../../../../x86_64-pc-linux-gnu/lib/../lib64/
/opt/gcc/7.2.0/lib/gcc/x86_64-pc-linux-gnu/7.2.0/../../../x86_64-pc-linux-gnu/7.2.0/
/opt/gcc/7.2.0/lib/gcc/x86_64-pc-linux-gnu/7.2.0/../../../x86_64-linux-gnu/
/opt/gcc/7.2.0/lib/gcc/x86_64-pc-linux-gnu/7.2.0/../../../../lib64/
/lib/x86_64-pc-linux-gnu/7.2.0/
/lib/x86_64-linux-gnu/
/lib/../lib64/
/usr/lib/x86_64-pc-linux-gnu/7.2.0/
/usr/lib/x86_64-linux-gnu/
/usr/lib/../lib64/
/tmp/XXX/lib/
/opt/gcc/7.2.0/lib/gcc/x86_64-pc-linux-gnu/7.2.0/../../../../x86_64-pc-linux-gnu/lib/
/opt/gcc/7.2.0/lib/gcc/x86_64-pc-linux-gnu/7.2.0/../../../
/lib/
/usr/lib/

In addition to the normal search priority - where LIBRARY_PATH comes before system paths, GCC prioritizes a few "prefixes", including ../lib64. This can be worked around by creating another symlink:

ln -s lib /tmp/XXX/lib64

I thought this was related to the --libdir parameter during configure, which I omitted and is /usr/lib in the system installation, but even if i specify --libdir=$PREFIX/lib --libexecdir=$PREFIX/lib, it prefers ../lib64.

How to compile or control gcc at runtime so that it at least uses the ../lib instead of ../lib64 postfix?

Clang

Clang is even more uncooperative. It does not include LIBRARY_PATH in the output for --print-search-dirs and does not even include a -L/tmp/XXX/lib to it's call to ld if the libfoo.so can be found already in /usr/lib.

How can I get Clang to prioritize my library path transparently?

Notes

Solution

  • In addition to the normal search priority - where LIBRARY_PATH comes before system paths, GCC prioritizes a few "prefixes", including ../lib64.

    This is not normal GCC behaviour, which is probably why it's also not the behaviour you see for Clang.

    As documented in the GCC manual, the directories specified by LIBRARY_PATH are passed to ld after the directories explicitly named with -L options:

    The value of LIBRARY_PATH is a colon-separated list of directories, much like PATH. When configured as a native compiler, GCC tries the directories thus specified when searching for special linker files, if it cannot find them using GCC_EXEC_PREFIX. Linking using GCC also uses these directories when searching for ordinary libraries for the -l option (but directories specified with -L come first).

    The behaviour you're seeing where these paths come first is not how GCC usually behaves:

    /tmp/XXX/lib/x86_64-pc-linux-gnu/7.2.0/
/tmp/XXX/lib/x86_64-linux-gnu/
/tmp/XXX/lib/../lib64/

    This seems to be caused by a patch to GCC used by Debian (and inherited by Ubuntu) as observed by @facetus. That patch seems to be intended to make GCC more compatible with the Debian MultiArch directory hierarchy, where libs are in an arch-specific sub-directory like /usr/lib/x86_64-linux-gnu rather than just in /usr/lib or /usr/lib64. As a side effect of those patches, you can "trick" gcc into looking in your preferred directories first with the right combination of symlinks that match the extra dirs Debian searches first.

    One of the comments suggests a cleaner solution would be to use GCC Spec Files. You could create a simple specs file that modifies the default *link spec and then tell gcc to use that via -specs=yourspecsfile e.g. something like:

    %rename link                 orig_link

*link:
-L/your/custom/libs %(orig_link)

    Alternatively, I think you could make that the default behaviour by configuring gcc with:

    --with-specs="%{m64:-L/your/custom/libs}"

    I've used other custom specs this way, although not this specific example. I think this should mean that every time your custom gcc is compiling (or assembling or linking) some 64-bit code, it will add the -L/your/custom/libs option. I think that should do what you want for a self-built GCC.

    But I have no idea how to get Clang to do something similar.