I am using cross-prelink to prelink a large C++ executables that use Qt for an embedded ARM device. Note that I am not using Yocto, but a custom distribution - so I am running prelink manually at the moment.
Looking at the output of prelink, it seems to work:
$ prelink --verbose --ld-library-path=/opt/<product>/lib:/usr/local/Qt-5.3.1/lib --root=$PRODUCT_TARGET_ROOT/<product>/rfs/ /path/to/binary
Laying out 56 libraries in virtual address space 41000000-50000000
Assigned virtual address space slots for libraries:
/lib/ld-linux.so.3 41000000-41027908
/opt/<product>/lib/lib<product>common.so.1 41030000-41cf0fd0
/lib/libc.so.6 442b0000-443e3980
/usr/local/Qt-5.3.1/lib/libQt5Qml.so.5 434f0000-4380ee84
[..]
Prelinking /lib/ld-2.17.so
Prelinking /lib/libc-2.17.so
Prelinking /path/to/binary
Prelinking /<product>/lib/lib<product>common.so.1.0.0
Prelinking /usr/local/Qt-5.3.1/lib/libQt5Qml.so.5.3.1
[..]
When the library gets loaded, at least libQt5Qml.so and libproductcommon.so seem to get loaded to the preferred load address set by prelink:
$ cat /proc/`pidof binary`/maps
2ab49000-2ab4a000 r--p 0001e000 07:00 9357 /roroot/lib/ld-2.17.so
2ab4a000-2ab4b000 rw-p 0001f000 07:00 9357 /roroot/lib/ld-2.17.so
2b0fd000-2b223000 r-xp 00000000 07:00 9730 /roroot/lib/libc-2.17.so
2b223000-2b22a000 ---p 00126000 07:00 9730 /roroot/lib/libc-2.17.so
2b22a000-2b22c000 r--p 00125000 07:00 9730 /roroot/lib/libc-2.17.so
2b22c000-2b22d000 rw-p 00127000 07:00 9730 /roroot/lib/libc-2.17.so
41030000-41ce7000 r-xp 00000000 07:00 9305 /roroot/<product>/lib/lib<product>common.so.1.0.0
41ce7000-41cef000 ---p 00cb7000 07:00 9305 /roroot/<product>/lib/lib<product>common.so.1.0.0
41cef000-41cf1000 rw-p 00cb7000 07:00 9305 /roroot/<product>/lib/lib<product>common.so.1.0.0
434f0000-437f8000 r-xp 00000000 07:00 1355 /roroot/usr/local/Qt-5.3.1/lib/libQt5Qml.so.5.3.1
437f8000-437ff000 ---p 00308000 07:00 1355 /roroot/usr/local/Qt-5.3.1/lib/libQt5Qml.so.5.3.1
437ff000-4380e000 rw-p 00307000 07:00 1355 /roroot/usr/local/Qt-5.3.1/lib/libQt5Qml.so.5.3.1
[..]
Now, I expected to see some reductions in the number of relocations:
$ LD_DEBUG=statistics /path/to/binary
20453: number of relocations: 66379
20453: number of relocations from cache: 38995
20453: number of relative relocations: 21690
$ LD_USE_LOAD_BIAS=0 LD_DEBUG=statistics /path/to/binary
20478: number of relocations: 66379
20478: number of relocations from cache: 38995
20478: number of relative relocations: 62981
This shows that only the relative relocations were reduced due to prelink, but not the normal relocations (that presumably need a symbol lookup). I am especially interested to reduce the other relocations, since those are presumably the more expensive ones.
Now my questions:
OK, it turned out the problem was that some libraries were not correctly prelinked, as seen in my original question, in which e.g. libc.so wasn't loaded at the correct load address.
Seems like prelinking is a all-or-nothing approach: If one of the dependencies of an executable isn't correctly prelinked or can't be loaded at the preferred address, then neither the executable nor the libraries will be able to take advantage of prelinked symbol relocations, and only take advantage of prelinked relative relocations.
Whether a library was correctly prelinked should, in addition to the above, be checked with:
# readelf --dynamic usr/lib/someLibrary.so
[..]
0x6ffffdf5 (GNU_PRELINKED) 2014-12-15T14:16:56
[..]
# readelf --program-headers usr/lib/someLibrary.so
Program Headers:
Type Offset VirtAddr PhysAddr FileSiz MemSiz Flg Align
[..]
LOAD 0x000000 0x44bf0000 0x44bf0000 0xb56d4 0xb56d4 R E 0x8000
[..]
The address outputted by prelink --verbose, readelf --program-headers and cat /proc/PID/maps need to match.
My mistake was that I didn't check readelf - if I had done so, I would have realized that some libraries on the target device were not prelinked, because an error in the buildsystem caused the prelinked versions to be overwritten with the non-prelinked ones...
After fixing my buildsystem problem, the normal relocations indeed went down to 0:
# LD_DEBUG=statistics /path/to/binary
5089: number of relocations: 0
5089: number of relocations from cache: 19477
5089: number of relative relocations: 0