Search code examples
c++linuxgdb

How to debug process loaded by dynamic loader manually with gdb

With a simple program:

> cat ./helloworld.cc
#include <iostream>
#include <thread>
#include <chrono>

int main() {
    while (true) {
        std::cout << "Hello World\n";
        std::this_thread::sleep_for(std::chrono::seconds(1));
    }
}

and compiled like this:

> g++ -Wall helloworld.cc -o helloworld -g

And run by manually loading it using ld.so:

> /lib64/ld-linux-x86-64.so.2  ./helloworld
Hello World
Hello World
Hello World
Hello World

Suppose this process's pid (process id) is 12345, and I attach to it using gdb:

> sudo gdb -p 12345
GNU gdb (Ubuntu 12.1-0ubuntu1~22.04) 12.1
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word".
Attaching to process 83860
Reading symbols from /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2...
Reading symbols from /usr/lib/debug/.build-id/61/ef896a699bb1c2e4e231642b2e1688b2f1a61e.debug...
Reading symbols from /lib/x86_64-linux-gnu/libstdc++.so.6...
(No debugging symbols found in /lib/x86_64-linux-gnu/libstdc++.so.6)
Reading symbols from /lib/x86_64-linux-gnu/libc.so.6...
Reading symbols from /usr/lib/debug/.build-id/69/389d485a9793dbe873f0ea2c93e02efaa9aa3d.debug...
Reading symbols from /lib/x86_64-linux-gnu/libm.so.6...
Reading symbols from /usr/lib/debug/.build-id/27/e82301dba6c3f644404d504e1bb1c97894b433.debug...
Reading symbols from /lib64/ld-linux-x86-64.so.2...
Reading symbols from /usr/lib/debug/.build-id/61/ef896a699bb1c2e4e231642b2e1688b2f1a61e.debug...
Reading symbols from /lib/x86_64-linux-gnu/libgcc_s.so.1...
(No debugging symbols found in /lib/x86_64-linux-gnu/libgcc_s.so.1)
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007f2b474e57fa in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=0x7ffc1ea7ff20, rem=0x7ffc1ea7ff20) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78
78      ../sysdeps/unix/sysv/linux/clock_nanosleep.c: No such file or directory.
(gdb) bt
#0  0x00007f2b474e57fa in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=0x7ffc1ea7ff20, rem=0x7ffc1ea7ff20)
    at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78
#1  0x00007f2b474ea6e7 in __GI___nanosleep (req=<optimized out>, rem=<optimized out>) at ../sysdeps/unix/sysv/linux/nanosleep.c:25
#2  0x00007f2b47b0a54f in ?? ()      
#3  0x00007f2b47b0d040 in ?? ()    # <=== Missing stack information
#4  0x00007ffc1ea7ff60 in ?? ()
#5  0x0000000000000001 in ?? ()
#6  0x0000000000000000 in ?? ()
(gdb)

So, why is the stack info missed, and how could I debug it using gdb ? I am running on Ubuntu22.04, with gcc 11.3.0, gdb 12.1.

If I run the "./helloworld" program normally, then the stack info is completed and normal:

> sudo gdb -p 12345
GNU gdb (Ubuntu 12.1-0ubuntu1~22.04) 12.1
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word".
Attaching to process 83928
Reading symbols from /tmp/helloworld...
Reading symbols from /lib/x86_64-linux-gnu/libstdc++.so.6...
(No debugging symbols found in /lib/x86_64-linux-gnu/libstdc++.so.6)
Reading symbols from /lib/x86_64-linux-gnu/libc.so.6...
Reading symbols from /usr/lib/debug/.build-id/69/389d485a9793dbe873f0ea2c93e02efaa9aa3d.debug...
Reading symbols from /lib/x86_64-linux-gnu/libm.so.6...
Reading symbols from /usr/lib/debug/.build-id/27/e82301dba6c3f644404d504e1bb1c97894b433.debug...
Reading symbols from /lib64/ld-linux-x86-64.so.2...
Reading symbols from /usr/lib/debug/.build-id/61/ef896a699bb1c2e4e231642b2e1688b2f1a61e.debug...
Reading symbols from /lib/x86_64-linux-gnu/libgcc_s.so.1...
(No debugging symbols found in /lib/x86_64-linux-gnu/libgcc_s.so.1)
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007f543c2e57fa in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=0x7ffdf8c83640, rem=0x7ffdf8c83640) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78
78      ../sysdeps/unix/sysv/linux/clock_nanosleep.c: No such file or directory.
(gdb) bt
#0  0x00007f543c2e57fa in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=0x7ffdf8c83640, rem=0x7ffdf8c83640)
    at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78
#1  0x00007f543c2ea6e7 in __GI___nanosleep (req=<optimized out>, rem=<optimized out>) at ../sysdeps/unix/sysv/linux/nanosleep.c:25
#2  0x000056445831054f in std::this_thread::sleep_for<long, std::ratio<1l, 1l> > (__rtime=...) at /usr/include/c++/11/bits/this_thread_sleep.h:82
#3  0x0000564458310243 in main () at helloworld.cc:8
(gdb)
Solution

  • So, why is the stack info missed

    Because GDB looks at /proc/$pid/exe symbolic link and decides that the main binary is /lib64/ld-linux-x86-64.so.2, but that is not the case here.

    how could I debug it using gdb ?

    The easiest way is to simply not do that. Instead run the binary normally, so GDB uses the correct executable. See also http://xyproblem.info.

    But if you must debug a process started this way, there is a way to do so. You just need to tell GDB what symbol file it should use.

    Example:

    /lib64/ld-linux-x86-64.so.2 /bin/sleep 3600 &
[1] 31

gdb -p 31
GNU gdb (GDB) Fedora Linux 13.2-2.fc37
...
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
__GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=0x7ffccc885bb0, rem=0x7ffccc885ba0)
    at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:71
Downloading source file /usr/src/debug/glibc-2.36-11.fc37.x86_64/time/../sysdeps/unix/sysv/linux/clock_nanosleep.c
71        return -r;
(gdb) bt
#0  __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=0x7ffccc885bb0, rem=0x7ffccc885ba0)
    at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:71
#1  0x00007f3d381957f7 in __GI___nanosleep (req=<optimized out>, rem=<optimized out>)
    at ../sysdeps/unix/sysv/linux/nanosleep.c:25
#2  0x00007f3d382a79e0 in ?? ()
#3  0x40ac200000000000 in ?? ()
#4  0x0000000000000000 in ?? ()

    This is the same output you got. The next step is to figure out where the actual binary is loaded:

    grep sleep /proc/31/maps | head -1
7f3d382a5000-7f3d382a7000 r--p 00000000 08:20 10725         /usr/bin/sleep

    We want to add symbols for /usr/bin/sleep loaded @0x7f3d382a5000. Unfortunately, GDB add-symbol-file command doesn't want the 0x7f3d382a5000 "load" address, it wants the address where .text begins.

    readelf -WS  /usr/bin/sleep | grep '\.text'
  [16] .text             PROGBITS        00000000000025d0 0025d0 002e12 00  AX  0   0 16

    Now we have all the pieces we need:

    (gdb) add-symbol-file /usr/bin/sleep 0x25d0+0x7f3d382a5000
add symbol table from file "/usr/bin/sleep" at
        .text_addr = 0x7f3d382a75d0
(y or n) y
Reading symbols from /usr/bin/sleep...

(gdb) bt
#0  __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x7ffccc885bb0, rem=rem@entry=0x7ffccc885ba0)
    at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:71
#1  0x00007f3d381957f7 in __GI___nanosleep (req=req@entry=0x7ffccc885bb0, rem=rem@entry=0x7ffccc885ba0)
    at ../sysdeps/unix/sysv/linux/nanosleep.c:25
#2  0x00007f3d382a79e0 in rpl_nanosleep (remaining_delay=0x7ffccc885ba0, requested_delay=0x7ffccc885ba0) at ../lib/nanosleep.c:83
#3  xnanosleep (seconds=<optimized out>) at ../lib/xnanosleep.c:69
#4  main (argc=<optimized out>, argv=<optimized out>) at ../src/sleep.c:142

    QED.