I have an executable which depends on two basic boost libraries, libboost_system and libboost_thread, and when the executable is loading the libraries, the search path bafflingly differs with respect to lib/tls:
$ LD_DEBUG=libs ./var.exe
25130: find library=libboost_system.so.1.55.0 [0]; searching
25130: search path=/opt/boost_1_55_0/stage/lib/tls/x86_64:/opt/boost_1_55_0/stage/lib/tls:/opt/boost_1_55_0/stage/lib/x86_64:/opt/boost_1_55_0/stage/lib (RPATH from file ./var.exe)
25130: trying file=/opt/boost_1_55_0/stage/lib/tls/x86_64/libboost_system.so.1.55.0
25130: trying file=/opt/boost_1_55_0/stage/lib/tls/libboost_system.so.1.55.0
25130: trying file=/opt/boost_1_55_0/stage/lib/x86_64/libboost_system.so.1.55.0
25130: trying file=/opt/boost_1_55_0/stage/lib/libboost_system.so.1.55.0
25130:
25130: find library=libboost_thread.so.1.55.0 [0]; searching
25130: search path=/opt/boost_1_55_0/stage/lib (RPATH from file ./var.exe)
25130: trying file=/opt/boost_1_55_0/stage/lib/libboost_thread.so.1.55.0
The executable has been linked with exactly the same -rpath=/opt/... setting for both libraries, and I know that boost have been built by a common invocation of b2 (ie exactly the same command line parameters for both). The /etc/ld.so.cache doesn't have any related entries.
Further, it doesn't seem that there is anything remarkable about the built libraries themselves in terms of rpath or ABI requirements,
$ readelf -d /opt/boost_1_55_0/stage/lib/libboost_system.so.1.55.0 | grep -i rpath
# empty
$ eu-readelf -h /opt/boost_1_55_0/stage/lib/libboost_system.so.1.55.0
# there is no difference in ABI versions
Both libraries have exactly the same shared library dependencies, apart from the fact that libboost_thread depends on libboost_system, and both need GLIBC_2.2.5.
I reckon that it is decided that libboost_system somehow needs to links with an NPTL glibc and that's why <rpath>/lib/tls is searched, but it isn't obvious to me looking at the objdump of the libboost_system library why it is the case.
How is an NPTL library is tagged? How does dlopen(3) decide to look in the lib/tls path?
the search path bafflingly differs with respect to lib/tls
You are barking up the wrong tree.
The GLIBC loader takes RPATH from your executable, and appends PLATFORM strings (tls/x86_64:tls:x86_64 here) to it to construct the full search path.
Once the loader discovers that e.g. RPATH/tls directory does not exist, it removes that entry from the search path (there is no point in repeatedly searching non-existent directory). Eventually you end up with just RPATH itself being present.
If you relink your binary to e.g. invert the order of dependency on libboost_thread and libboost_system, you'll discover that again only the first library is being searched in RPATH/tls, etc. and the second only in RPATH.