I've a C++ application running on a remote server. I recently introduced a lot of new code. Fearing crashes, I had set ulimit -c unlimited and some time later I got a crash, with a coredump, which helped me debug a problem. For business reasons, the running binary has no debug symbols, but I do have the with-symbols binary on my PC, so debugging was a breeze.
Today the updated service crashed again, unfortunately it didn't produce a coredump this time (the old core file was still there, untouched, I guess it may be some kind of expected behaviour). The crash was within realloc() innards this time, so it presented me with this stack trace to stdout:
*** Error in `./MyApp': corrupted double-linked list: 0x0000000003a04940 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7f05ed2897e5]
/lib/x86_64-linux-gnu/libc.so.6(+0x7e6ed)[0x7f05ed2906ed]
/lib/x86_64-linux-gnu/libc.so.6(+0x81cde)[0x7f05ed293cde]
/lib/x86_64-linux-gnu/libc.so.6(__libc_malloc+0x54)[0x7f05ed296184]
/lib/x86_64-linux-gnu/libc.so.6(realloc+0x358)[0x7f05ed296a18]
./MyApp[0x453f58]
./MyApp[0x454a42]
./MyApp[0x457cd6]
./MyApp[0x45eb19]
./MyApp[0x49cfd7]
./MyApp[0x49707b]
./MyApp[0x70734e]
...
a lot more lines
...
./MyApp[0x664c65]
./MyApp[0x73e7b2]
./MyApp[0x70d849]
./MyApp[0x783af4]
./MyApp[0x425da8]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7f05ed232830]
./MyApp[0x43a0c9]
======= Memory map: ========
...
< a huge table of memory mappings, ending with: >
Aborted (core dumped)
As stated above, the core file is not changed from the previous crash, so it cannot be used.
I was wondering whether I could use this stack trace to manually find out which function triggered the realloc() that wrecked everything. I tried addr2line using the addresses mentioned, but I feel it sends me to the wrong places, as they are completely irrelevant. Probably I should use the memory map in some way I don't understand and couldn't find out after some googling. Is there a guide for using this type of stack traces?
objdump - one cool program from GNU toolchain, that can show you information about binary. linked libraries, memory alignments, function tables and much more.
Common use:
objdump -T <file>
There are some more tools, that can help you. like nm or readelf (for elf files).
nm -g -C <file>
readelf -sW <file>