Using dlsym() to stub malloc/free leads to segmentation fault

Question

I started to dabble in unit testing C code (using check) and stubbing functions. I am trying to unit test a small library of data structures that I wrote and wanted to test how it would react to OOM. So I wrote a simple stubs.c file containing:

#include <stdlib.h>
#include <errno.h>
#include <dlfcn.h>

static int malloc_fail_code = 0;
static int calloc_fail_code = 0;

void set_malloc_fail_code(int no) { malloc_fail_code = no; }
void set_calloc_fail_code(int no) { calloc_fail_code = no; }

void *malloc(size_t size)
{
    static void *(*real_malloc)(size_t) = NULL;

    if (!real_malloc)
        real_malloc = (void *(*)(size_t)) dlsym(RTLD_NEXT, "malloc");

    if (malloc_fail_code != 0) {
        errno = malloc_fail_code;
        malloc_fail_code = 0;
        return NULL;
    }

    return real_malloc(size);
}

void *calloc(size_t nmemb, size_t size)
{
    static void *(*real_calloc)(size_t, size_t) = NULL;

    if (!real_calloc)
        real_calloc = (void *(*)(size_t, size_t)) dlsym(RTLD_NEXT, "calloc");

    if (calloc_fail_code != 0) {
        errno = calloc_fail_code;
        calloc_fail_code = 0;
        return NULL;
    }

    return real_calloc(nmemb, size);
}

with its relative stubs.h containing the definitions for the two setters. I then compiled stubs.c as a shared object called libstubs.so. I also compiled my library as a shared object called libmy_lib.so.

My test code is in test.c is something like this:

#include <stdlib.h>
#include <errno.h>
#include <check.h>

#include "my_lib.h"
#include "stubs.h"

START_TEST(my_test)
{
    ... // using the two setters I force malloc and calloc to return null and set errno to ENOMEM
}
END_TEST

... // check boilerplate to create suite and add tests

I then linked the test executable against libmy_lib.so and libstubs.so. Running said executable greets me with a segfault. Inspecting the crash with gdb makes me believe that I encountered a stack overflow due to infinte recursion (gdb backtrace):

#0  0x00007ffff7fc143c in calloc (
    nmemb=<error reading variable: Cannot access memory at address 0x7fffff7feff8>, 
    size=<error reading variable: Cannot access memory at address 0x7fffff7feff0>)
    at stubs.c
#1  0x00007ffff7db9c88 in _dlerror_run (operate=operate@entry=0x7ffff7db94f0 <dlsym_doit>, 
    args=args@entry=0x7fffff7ff030) at dlerror.c:148
#2  0x00007ffff7db9570 in __dlsym (handle=<optimized out>, name=<optimized out>) at dlsym.c:70
#3  0x00007ffff7fc1487 in calloc (nmemb=1, size=32) at stubs.c
...

I tried including directly stubs.c into test.c but no luck. I also tried writing a small unit testing framework of my own that extends stubs.c and it worked. However I don't want to waste time reinventing the wheel and I am sure there is something I am doing wrong in linking since I don't know much in compilation/linking.

For compilation I am using the meson build system so I don't know how to get the exact command line arguments but I can write a MWE of my build targets:

lib = library(
  'my_lib',
  sources,
  include_directories: includes,
  install: true
)

stubs = shared_library(
  'stubs',
  'stubs.c',
  c_args: ['-g'],
  include_directories: test_includes,
  link_args: ['-ldl']
)

test_exe = executable(
  'test_exe',
  c_args: ['-g'],
  sources: 'test.c',
  dependencies: check,
  link_with: [stubs, lib],
  include_directories: includes + test_includes
)
test('test', test_exe, suite: 'suite')

Answer 1

Try using LD_PRELOAD trick. The meson-ish way to accomplish it would be:

  test_env = environment()
  test_env.prepend('LD_PRELOAD', stubs.full_path())
  test('test', test_exe, suite: 'suite', env: test_env)

note: do not link executable with stubs.