Search code examples
linuxmalloclibcdlopen

overwrite malloc/free in dlopen-ed library

I have a shared library - plugin.so, which is dlopen-ed by the host program with flag RTLD_LOCAL, I have my own memory operation functions defined in that library:

void *plugin_malloc(size_t size) { /* ... */ }
void plugin_free(void *ptr) { /* ... */ }

What I need is to replace ALL malloc/free calls in plugin.so with my own plugin_malloc/plugin_free, I tried using GCC's alias attribute extension:

void *malloc(size_t) __attribute__((alias("plugin_malloc"), used))
void free(void*) __attribute__((alias("plugin_free"), used))

However, this only works when the library is linked into the host program, but not work with the dlopen way.

I'm on Linux with compiler GCC-4.8.5, and I have the source code of plugin.so and can modify it as I like, but I cannot modify the host program, and replace malloc/free not only in plugin.so but also in entire program is also acceptable.

So, is there any solution? Thanks.

EDIT: I also have no permission to modify the host program's startup arguments, environment variables, what I can do is just providing the plugin.so to guys who own the host program, and they run the host program and dlopen my plugin.so.

Solution

  • What I need is to replace ALL malloc/free calls in plugin.so with my own plugin_malloc/plugin_free,

    This is trivial to do.

    Suppose you have foo.o and plugin_malloc.o that are linked into plugin.so. The foo.o uses malloc and free, while plugin.o defines plugin_malloc and plugin_free.

    Then:

    objcopy --redefine-sym malloc=plugin_malloc --redefine-sym free=plugin_free foo.o foo2.o
gcc -shared -fPIC plugin.o foo2.o -o plugin.so

    Voila: all references to malloc and free in foo.o have been replaced. Enjoy.

    Update:

    if I call a glibc function which allocates memory and needs to be freed in my code, the program crashes. e.g. char *s = strdup("hello"); free(s); because strdup calls glibc's malloc, but the later free is my plugin_free

    There are a few ways, only some of which satisfy your other constraints:

    1. you must replace all mallocs and frees with your own for the entire program (e.g. via LD_PRELOAD, or by statically linking malloc implementation into the main executable), or
    2. you must ensure that you don't call any function that allocates memory via malloc that you expect to free later, or
    3. for any such calls to stdup or asprintf etc., when you want to deallocate this memory, you must call __libc_free rather than plugin_free, or
    4. in plugin_malloc, pad all memory you allocate with 16 extra byte header (for alignment) write magic number to the start of the block, and return a pointer past the header to the caller. In plugin_free, check for proper alignment, then check the header for magic number. If it's there, subtract 16 from the pointer and use the rest of plugin_free to free the memory. If the magic number is not there, assume the pointer didn't come from plugin_malloc, and call __libc_free on it instead.
    5. In plugin_malloc keep track of all the blocks you've ever allocated. In plugin_free check that list, and use __libc_free if the pointer is not on the list.

    Since you have all sources for your plugin, either solution 2 or 3 should be doable, but obviously requires that you audit every call to free to see where the memory came from.

    Variants 4 and 5 don't require such audit, but don't cleanly separate memory allocated by plugin from that allocated by the main program.

    Don't forget about other ways to allocate memory: realloc, memalign, posix_memalign, etc.