backtrace() falls into endless recursion?

Question

I use backtrace() function in one self-defined malloc(), like this:

    void *malloc(size_t size)
    {printf("my malloc!\n");
            //map_insert

        static void *(*mallocp)(size_t size);
        char *error;
        void *ptr;
        void *buffer[100];
        int nptrs;
        char ptrs_num[10];
        memset(ptrs_num, '\0', 10);
        char **strings;
        printf("1\n");
        nptrs = backtrace(buffer, 100);
/* those code below seems useless because problem happens here*/
        printf("2\n");
        printf("backtrace() returned %d addresses\n", nptrs);
        printf("3\n");
        strings = backtrace_symbols(buffer, nptrs);
        printf("4\n");
        if (strings == NULL) {
           perror("backtrace_symbols");
           exit(1);
        }
        /*code neglected*/
        return ptr;
    }

This is part of code in my .so file, and some bug may exist because I haven't run this and modify. I use this test code:

#include <stdlib.h>
#include <malloc.h>
#include <stdio.h>
int main(int argc, char **argv){

    printf("to malloc a!\n");
    char *a=(char *)malloc(sizeof(int)*1024);
    return 0;
}

my result is as below:

my malloc!
1
my malloc!
1
my malloc!
1
my malloc!
1
my malloc!
1
my malloc!
1
my malloc!
1
my malloc!
1
my malloc!
1
my malloc!
1
my malloc!
1
Segmentation fault

I really don't know why is this.

I think backtrace would just get some information, why it would trigger my malloc?

Answer 1

backtrace() calls malloc(). There's a recursive call to your hook malloc funcion. This is a known issue. In order to use backtrace() from with malloc(), you can setup a flag that indicates it was called from within your hooked function and if so, allocate and return the memory requested by using a different mechanism other than malloc() such as using mmap(), brk() etc.

Also see: https://sourceware.org/ml/libc-alpha/2015-02/msg00653.html