I'm using a wrapper library to trace functions using LD_PRELOAD which does work when the functions I'm tracing are referenced in the application.
The wrapper library uses dlsym to populate symbols it wraps.
But this is not working if the application doesn't reference functions directly but through dlopen.
Should the wrapper library work with dynamically loaded libraries? If not, is there a way to make it work?
But this is not working if the application doesn't reference functions directly but through dlopen.
If the application performs:
void *h = dlopen("libfoo.so", ...);
void *sym = dlsym(h, "symbol");
then the symbol will be resolved to libfoo.so, regardless of any LD_PRELOADs (and indeed regardless of any other instances of symbol in other already loaded libraries). This is working as intended.
Should the wrapper library work with dynamically loaded libraries?
No.
If not, is there a way to make it work?
Yes, you can make it work. You would need to provide a replacement.so which provides all the symbols which the application looks up in libfoo.so, and then make it so the replacement.so is dlopend()ed by the application.
One way to do that is to rename libfoo.so -> libfoo.so.orig, copy replacement.so -> libfoo.so, and have replacement.so itself dlopen("libfoo.so.orig", ...).
If the application dlopens libfoo.so without absolute path, you may be able to arrange so replacement.so is earlier in the search path, e.g.
mkdir /tmp/replacement
ln -s /path/to/replacement.so /tmp/replacement/liboo.so
LD_LIBRARY_PATH=/tmp/replacement /path/to/a.out
If you go that route, replacement.so would still need to know how to dlopen the original libfoo.so (could use hard-coded absolute path for that).