Search code examples
cmacosmacos-mojavedlsym

blocking call to `dlsym` (RTLD_NEXT)

I am trying to wrap function. To do that I thought about using dlsym to get the real function location in memory after having injected my dynamic library.

I am running on MacOS Mojave (10.14.6) with Apple clang version 11.0.0 (clang-1100.0.33.8).

I tried to run a rather simple code displaying a message each time malloc was called.

So I used the code of this post : Overriding 'malloc' using the LD_PRELOAD mechanism

My test code was :

#include <stdlib.h>

int main()
{
    (void)malloc(1);
    (void)malloc(1);
    return (0);
}

I injected the dynamic library using DYLD_INSERT_LIBRARIES=./mylib.so DYLD_FORCE_FLAT_NAMESPACE=1 on OSX and LD_PRELOAD=./mylib.so on Ubuntu.

On OSX, the program is blocking on the dlsym call where as on a Ubuntu docker box, the program works fine.

EDIT:

As @R.. pointed out, the dlsym implementation on OSX is calling malloc (see sources : https://opensource.apple.com/source/cctools/cctools-667.8.0/libdyld/dlopen.c)

Solution

  • It's likely that calling dlsym causes a recursive call to malloc, which in turn causes a recursive call to dlsym, which deadlocks the second time because it already holds some lock. Using preload libraries to wrap malloc, rather than fully replacing it, is generally a bad idea, for this and other reasons.

    You might be able to use LD_AUDIT or the equivalent functionality on OSX (if any) to print a trace of when malloc is called. Otherwise you could drop in a full replacement malloc with debug output.