I would like to debug with gdb an app on an embedded system. This app requires a newer glibc, threading and works correctly by invoking the Linux dynamic loader ld-linux-x86-64.so.2. What I would like is to attach gdb and see the symbols and stack trace, but the loader "interferes" with gdb.
Here's a sample test.c file:
#include <stdio.h>
#include <unistd.h>
#include <stdlib.h>
#include <sys/types.h>
#include <pthread.h>
void *subThread(void *arg) {
printf("Thread started after '%d' chars.\n", (long *)arg);
sleep(2);
printf("Thread '%d' ended.\n", (long *)arg);
pthread_exit(NULL);
}
int main() {
pid_t pid = getpid();
if (pid > 0) {
printf("The process id is %d.\n", pid);
printf("Type 'q' to exit, 't' to spawn a thread.\n");
char readChar;
long count = 0;
do {
count++;
readChar = getchar();
if (readChar == 't') {
// Spawn a thread
pthread_t thread_id;
pthread_create(&thread_id, NULL, subThread, (void *) count);
pthread_join(thread_id, NULL);
}
} while(readChar != 'q');
printf("Main ended.\n");
return(0);
} else {
return(1);
}
}
I compile this source file in a system with a specific glibc version like so:
othersystem:~# gcc -O2 -g -lpthread -o test test.c
Then I move the test binary file into my embedded system, that has an older glibc version and that cannot run test directly, so I'm copying the loader and the required libraries too. See also my gdb settings.
[~/test] # ls
ld-linux-x86-64.so.2* libc.so.6* libpthread.so.0* libthread_db.so.1 test*
[~/test] # cat ~/.gdbinit
set auto-load safe-path /
set pagination off
set libthread-db-search-path /root/test
Now I run the code and press CTRLZ to pause and I want to attach gdb using the process id. The problem is that gdb sees the loader instead of the application and doesn't load the debug symbols.
[~/test] # /root/test/ld-linux-x86-64.so.2 --library-path /root/test /root/test/test
The process id is 30622.
Type 'q' to exit, 't' to spawn a thread.
^Z
[1]+ Stopped(SIGTSTP) /root/test/ld-linux-x86-64.so.2 --library-path /root/test /root/test/test
[~/test] # gdb /root/test/test 30622
GNU gdb (GDB) 10.1
Reading symbols from /root/test/test...
Attaching to program: /root/test/test, process 30622
warning: Build ID mismatch between current exec-file /root/test/test
and automatically determined exec-file /root/test/ld-linux-x86-64.so.2
exec-file-mismatch handling is currently "ask"
Load new symbol table from "/root/test/ld-linux-x86-64.so.2"? (y or n) y
Reading symbols from /root/test/ld-linux-x86-64.so.2...
(No debugging symbols found in /root/test/ld-linux-x86-64.so.2)
Reading symbols from /root/test/libpthread.so.0...
(No debugging symbols found in /root/test/libpthread.so.0)
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/root/test/libthread_db.so.1".
Reading symbols from /root/test/libc.so.6...
(No debugging symbols found in /root/test/libc.so.6)
Reading symbols from /root/test/ld-linux-x86-64.so.2...
(No debugging symbols found in /root/test/ld-linux-x86-64.so.2)
Reading symbols from /lib/libgcc_s.so.1...
0x00007fa1ef590494 in read () from /root/test/libc.so.6
(gdb) bt
#0 0x00007fa1ef590494 in read () from /root/test/libc.so.6
#1 0x00007fa1ef522670 in _IO_file_underflow () from /root/test/libc.so.6
#2 0x00007fa1ef5237b2 in _IO_default_uflow () from /root/test/libc.so.6
#3 0x00007fa1ef51de68 in getc () from /root/test/libc.so.6
#4 0x00007fa1ef68b110 in ?? ()
#5 0x00007fa1ef68b290 in ?? ()
#6 0x00007fa1ef4a2700 in ?? ()
#7 0x00007ffcdae50bb8 in ?? ()
#8 0x0000000000000000 in ?? ()
I do not know how to tell gdb to load symbols from /root/test/test; there are interesting articles here and here and also an answer on StackOveflow, but none worked... I'm not able to get the correct address to use the add-symbol-file directive.
I've tried like so:
[~/test] # cat /proc/30622/maps | grep "r-xp" | grep "/root/test/test" | awk -F'-' '{printf $1}'
7fa1ef68b000
[~/test] # objdump -s --section=".text" /root/test/test | grep Contents -A 1 | tail -n 1 | awk -F' ' '{printf $1}'
10c0
So the address should be 0x7fa1ef68b000 + 0x10c0 = 0x7fa1ef68c0c0, right? But when I type this into gdb, you can see that it doesn't pick up the debug symbols:
(gdb) add-symbol-file /root/test/test 0x7fa1ef68c0c0
add symbol table from file "/root/test/test" at
.text_addr = 0x7fa1ef68c0c0
(y or n) y
Reading symbols from /root/test/test...
(gdb) bt
#0 0x00007fa1ef590494 in read () from /root/test/libc.so.6
#1 0x00007fa1ef522670 in _IO_file_underflow () from /root/test/libc.so.6
#2 0x00007fa1ef5237b2 in _IO_default_uflow () from /root/test/libc.so.6
#3 0x00007fa1ef51de68 in getc () from /root/test/libc.so.6
#4 0x00007fa1ef68b110 in ?? ()
#5 0x00007fa1ef68b290 in ?? ()
#6 0x00007fa1ef4a2700 in ?? ()
#7 0x00007ffcdae50bb8 in ?? ()
#8 0x0000000000000000 in ?? ()
(gdb)
Is there a way to load the debug symbols without manually specifying the address? If I compile the binary to have the interpreter path hardcoded, like so:
othersystem:~# gcc -O2 -g -lpthread -Wl,-rpath /root/test -Wl,--dynamic-linker=/root/test/ld-linux-x86-64.so.2 -o test2 test.c
Then all is working right:
[~/test] # ldd /root/test/test2
linux-vdso.so.1 (0x00007ffcaa59c000)
libpthread.so.0 => /root/test/libpthread.so.0 (0x00007f7265e1d000)
libc.so.6 => /root/test/libc.so.6 (0x00007f7265c5c000)
/root/test/ld-linux-x86-64.so.2 (0x00007f7265e40000)
[~/test] # /root/test/test2
The process id is 23264.
Type 'q' to exit, 't' to spawn a thread.
^Z
[1]+ Stopped(SIGTSTP) ./test2
[~/test] # gdb /root/test/test2 23264
GNU gdb (GDB) 10.1
Reading symbols from /root/test/test2...
Attaching to program: /root/test/test2, process 23264
Reading symbols from /root/test/libpthread.so.0...
(No debugging symbols found in /root/test/libpthread.so.0)
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/root/test/libthread_db.so.1".
Reading symbols from /root/test/libc.so.6...
(No debugging symbols found in /root/test/libc.so.6)
Reading symbols from /root/test/ld-linux-x86-64.so.2...
(No debugging symbols found in /root/test/ld-linux-x86-64.so.2)
Program received signal SIGTSTP, Stopped (user).
0x00007ffba4e7d461 in read () from /root/test/libc.so.6
(gdb) bt
#0 0x00007ffba4e7d461 in read () from /root/test/libc.so.6
#1 0x00007ffba4e0f670 in _IO_file_underflow () from /root/test/libc.so.6
#2 0x00007ffba4e107b2 in _IO_default_uflow () from /root/test/libc.so.6
#3 0x00005613ef88a110 in getchar () at /usr/include/x86_64-linux-gnu/bits/stdio.h:49
#4 main () at test.c:24
As asked by Employed Russian, I'm adding the complete output of the maps and readelf (of course I had to use a new pid):
[~/test] # /root/test/ld-linux-x86-64.so.2 --library-path /root/test /root/test/test
The process id is 16873.
Type 'q' to exit, 't' to spawn a thread.
^Z
[1]+ Stopped(SIGTSTP) /root/test/ld-linux-x86-64.so.2 --library-path /root/test /root/test/test
[~/test] # cat /proc/16873/maps
7f5e3767b000-7f5e3767e000 rw-p 00000000 00:00 0
7f5e3767e000-7f5e376a0000 r--p 00000000 00:10 64323010 /root/test/libc.so.6
7f5e376a0000-7f5e377e8000 r-xp 00022000 00:10 64323010 /root/test/libc.so.6
7f5e377e8000-7f5e37834000 r--p 0016a000 00:10 64323010 /root/test/libc.so.6
7f5e37834000-7f5e37835000 ---p 001b6000 00:10 64323010 /root/test/libc.so.6
7f5e37835000-7f5e37839000 r--p 001b6000 00:10 64323010 /root/test/libc.so.6
7f5e37839000-7f5e3783b000 rw-p 001ba000 00:10 64323010 /root/test/libc.so.6
7f5e3783b000-7f5e3783f000 rw-p 00000000 00:00 0
7f5e3783f000-7f5e37845000 r--p 00000000 00:10 64323011 /root/test/libpthread.so.0
7f5e37845000-7f5e37854000 r-xp 00006000 00:10 64323011 /root/test/libpthread.so.0
7f5e37854000-7f5e3785a000 r--p 00015000 00:10 64323011 /root/test/libpthread.so.0
7f5e3785a000-7f5e3785b000 r--p 0001a000 00:10 64323011 /root/test/libpthread.so.0
7f5e3785b000-7f5e3785c000 rw-p 0001b000 00:10 64323011 /root/test/libpthread.so.0
7f5e3785c000-7f5e37862000 rw-p 00000000 00:00 0
7f5e37862000-7f5e37863000 r--p 00000000 00:10 64323013 /root/test/test
7f5e37863000-7f5e37864000 r-xp 00001000 00:10 64323013 /root/test/test
7f5e37864000-7f5e37865000 r--p 00002000 00:10 64323013 /root/test/test
7f5e37865000-7f5e37866000 r--p 00002000 00:10 64323013 /root/test/test
7f5e37866000-7f5e37867000 rw-p 00003000 00:10 64323013 /root/test/test
7f5e37867000-7f5e37868000 r--p 00000000 00:10 64323009 /root/test/ld-linux-x86-64.so.2
7f5e37868000-7f5e37886000 r-xp 00001000 00:10 64323009 /root/test/ld-linux-x86-64.so.2
7f5e37886000-7f5e3788e000 r--p 0001f000 00:10 64323009 /root/test/ld-linux-x86-64.so.2
7f5e3788e000-7f5e3788f000 r--p 00026000 00:10 64323009 /root/test/ld-linux-x86-64.so.2
7f5e3788f000-7f5e37890000 rw-p 00027000 00:10 64323009 /root/test/ld-linux-x86-64.so.2
7f5e37890000-7f5e37891000 rw-p 00000000 00:00 0
7f5e3905f000-7f5e39080000 rw-p 00000000 00:00 0 [heap]
7fff1fcd3000-7fff1fcf4000 rw-p 00000000 00:00 0 [stack]
7fff1fd82000-7fff1fd85000 r--p 00000000 00:00 0 [vvar]
7fff1fd85000-7fff1fd87000 r-xp 00000000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
[~/test] # readelf -Wl /root/test/test
Elf file type is DYN (Shared object file)
Entry point 0x1160
There are 11 program headers, starting at offset 64
Program Headers:
Type Offset VirtAddr PhysAddr FileSiz MemSiz Flg Align
PHDR 0x000040 0x0000000000000040 0x0000000000000040 0x000268 0x000268 R 0x8
INTERP 0x0002a8 0x00000000000002a8 0x00000000000002a8 0x00001c 0x00001c R 0x1
[Requesting program interpreter: /lib64/ld-linux-x86-64.so.2]
LOAD 0x000000 0x0000000000000000 0x0000000000000000 0x000768 0x000768 R 0x1000
LOAD 0x001000 0x0000000000001000 0x0000000000001000 0x0002fd 0x0002fd R E 0x1000
LOAD 0x002000 0x0000000000002000 0x0000000000002000 0x000210 0x000210 R 0x1000
LOAD 0x002dd8 0x0000000000003dd8 0x0000000000003dd8 0x000290 0x0002a8 RW 0x1000
DYNAMIC 0x002de8 0x0000000000003de8 0x0000000000003de8 0x0001f0 0x0001f0 RW 0x8
NOTE 0x0002c4 0x00000000000002c4 0x00000000000002c4 0x000044 0x000044 R 0x4
GNU_EH_FRAME 0x002098 0x0000000000002098 0x0000000000002098 0x000044 0x000044 R 0x4
GNU_STACK 0x000000 0x0000000000000000 0x0000000000000000 0x000000 0x000000 RW 0x10
GNU_RELRO 0x002dd8 0x0000000000003dd8 0x0000000000003dd8 0x000228 0x000228 R 0x1
Section to Segment mapping:
Segment Sections...
00
01 .interp
02 .interp .note.ABI-tag .note.gnu.build-id .gnu.hash .dynsym .dynstr .gnu.version .gnu.version_r .rela.dyn .rela.plt
03 .init .plt .plt.got .text .fini
04 .rodata .eh_frame_hdr .eh_frame
05 .init_array .fini_array .dynamic .got .got.plt .data .bss
06 .dynamic
07 .note.ABI-tag .note.gnu.build-id
08 .eh_frame_hdr
09
10 .init_array .fini_array .dynamic .got
[~/test] # readelf -WS /root/test/test
There are 37 section headers, starting at offset 0x4b18:
Section Headers:
[Nr] Name Type Address Off Size ES Flg Lk Inf Al
[ 0] NULL 0000000000000000 000000 000000 00 0 0 0
[ 1] .interp PROGBITS 00000000000002a8 0002a8 00001c 00 A 0 0 1
[ 2] .note.ABI-tag NOTE 00000000000002c4 0002c4 000020 00 A 0 0 4
[ 3] .note.gnu.build-id NOTE 00000000000002e4 0002e4 000024 00 A 0 0 4
[ 4] .gnu.hash GNU_HASH 0000000000000308 000308 000028 00 A 5 0 8
[ 5] .dynsym DYNSYM 0000000000000330 000330 000168 18 A 6 1 8
[ 6] .dynstr STRTAB 0000000000000498 000498 0000da 00 A 0 0 1
[ 7] .gnu.version VERSYM 0000000000000572 000572 00001e 02 A 5 0 2
[ 8] .gnu.version_r VERNEED 0000000000000590 000590 000040 00 A 6 2 8
[ 9] .rela.dyn RELA 00000000000005d0 0005d0 0000d8 18 A 5 0 8
[10] .rela.plt RELA 00000000000006a8 0006a8 0000c0 18 AI 5 23 8
[11] .init PROGBITS 0000000000001000 001000 000017 00 AX 0 0 4
[12] .plt PROGBITS 0000000000001020 001020 000090 10 AX 0 0 16
[13] .plt.got PROGBITS 00000000000010b0 0010b0 000008 08 AX 0 0 8
[14] .text PROGBITS 00000000000010c0 0010c0 000231 00 AX 0 0 16
[15] .fini PROGBITS 00000000000012f4 0012f4 000009 00 AX 0 0 4
[16] .rodata PROGBITS 0000000000002000 002000 000097 00 A 0 0 8
[17] .eh_frame_hdr PROGBITS 0000000000002098 002098 000044 00 A 0 0 4
[18] .eh_frame PROGBITS 00000000000020e0 0020e0 000130 00 A 0 0 8
[19] .init_array INIT_ARRAY 0000000000003dd8 002dd8 000008 08 WA 0 0 8
[20] .fini_array FINI_ARRAY 0000000000003de0 002de0 000008 08 WA 0 0 8
[21] .dynamic DYNAMIC 0000000000003de8 002de8 0001f0 10 WA 6 0 8
[22] .got PROGBITS 0000000000003fd8 002fd8 000028 08 WA 0 0 8
[23] .got.plt PROGBITS 0000000000004000 003000 000058 08 WA 0 0 8
[24] .data PROGBITS 0000000000004058 003058 000010 00 WA 0 0 8
[25] .bss NOBITS 0000000000004070 003068 000010 00 WA 0 0 16
[26] .comment PROGBITS 0000000000000000 003068 00001c 01 MS 0 0 1
[27] .debug_aranges PROGBITS 0000000000000000 003084 000040 00 0 0 1
[28] .debug_info PROGBITS 0000000000000000 0030c4 000626 00 0 0 1
[29] .debug_abbrev PROGBITS 0000000000000000 0036ea 000200 00 0 0 1
[30] .debug_line PROGBITS 0000000000000000 0038ea 00020c 00 0 0 1
[31] .debug_str PROGBITS 0000000000000000 003af6 000308 01 MS 0 0 1
[32] .debug_loc PROGBITS 0000000000000000 003dfe 0000e8 00 0 0 1
[33] .debug_ranges PROGBITS 0000000000000000 003ee6 000090 00 0 0 1
[34] .symtab SYMTAB 0000000000000000 003f78 000780 18 35 52 8
[35] .strtab STRTAB 0000000000000000 0046f8 0002bc 00 0 0 1
[36] .shstrtab STRTAB 0000000000000000 0049b4 000160 00 0 0 1
Key to Flags:
W (write), A (alloc), X (execute), M (merge), S (strings), I (info),
L (link order), O (extra OS processing required), G (group), T (TLS),
C (compressed), x (unknown), o (OS specific), E (exclude),
l (large), p (processor specific)
For an easier (future) reading, I'm reporting here the output after Employed Russian's solution.
[~/test] # /root/test/ld-linux-x86-64.so.2 --library-path /root/test /root/test/test
The process id is 16873.
Type 'q' to exit, 't' to spawn a thread.
^Z
[1]+ Stopped(SIGTSTP) /root/test/ld-linux-x86-64.so.2 --library-path /root/test /root/test/test
[~/test] # cat /proc/16873/maps | grep "/root/test/test" | awk -F'-' '{print "0x" $1; exit; }'
0x7f5e37862000
[~/test] # readelf -WS /root/test/test | grep '\.text ' | awk '{print "0x" $5}'
0x0010c0
[~/test] # gdb /root/test/test 16873
GNU gdb (GDB) 10.1
Reading symbols from /root/test/test...
Attaching to program: /root/test/test, process 16873
warning: Build ID mismatch between current exec-file /root/test/test
and automatically determined exec-file /root/test/ld-linux-x86-64.so.2
exec-file-mismatch handling is currently "ask"
Load new symbol table from "/root/test/ld-linux-x86-64.so.2"? (y or n) y
Reading symbols from /root/test/ld-linux-x86-64.so.2...
(No debugging symbols found in /root/test/ld-linux-x86-64.so.2)
Reading symbols from /root/test/libpthread.so.0...
(No debugging symbols found in /root/test/libpthread.so.0)
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/root/test/libthread_db.so.1".
Reading symbols from /root/test/libc.so.6...
(No debugging symbols found in /root/test/libc.so.6)
Reading symbols from /root/test/ld-linux-x86-64.so.2...
(No debugging symbols found in /root/test/ld-linux-x86-64.so.2)
Program received signal SIGTTIN, Stopped (tty input).
0x00007f5e37768461 in read () from /root/test/libc.so.6
(gdb) bt
#0 0x00007f5e37768461 in read () from /root/test/libc.so.6
#1 0x00007f5e376fa670 in _IO_file_underflow () from /root/test/libc.so.6
#2 0x00007f5e376fb7b2 in _IO_default_uflow () from /root/test/libc.so.6
#3 0x00007f5e37863110 in ?? ()
#4 0x00007f5e37863290 in ?? ()
#5 0x00007f5e37863160 in ?? ()
#6 0x00007fff1fcf2578 in ?? ()
#7 0x0000000000000000 in ?? ()
(gdb) add-symbol-file /root/test/test 0x7f5e37862000+0x0010c0
add symbol table from file "/root/test/test" at
.text_addr = 0x7f5e378630c0
(y or n) y
Reading symbols from /root/test/test...
(gdb) bt
#0 0x00007f5e37768461 in read () from /root/test/libc.so.6
#1 0x00007f5e376fa670 in _IO_file_underflow () from /root/test/libc.so.6
#2 0x00007f5e376fb7b2 in _IO_default_uflow () from /root/test/libc.so.6
#3 0x00007f5e37863110 in getchar () at /usr/include/x86_64-linux-gnu/bits/stdio.h:49
#4 main () at test.c:24
#5 0x00007f5e376a209b in __libc_start_main () from /root/test/libc.so.6
#6 0x00007f5e3786318a in _start ()
(gdb) q
A debugging session is active.
Inferior 1 [process 16873] will be detached.
Quit anyway? (y or n) y
Detaching from program: /root/test/ld-linux-x86-64.so.2, process 16873
[Inferior 1 (process 16873) detached]
[1]+ Stopped(SIGTTIN) /root/test/ld-linux-x86-64.so.2 --library-path /root/test /root/test/test
So I have to reply Y to the request of the file mismatch and then add-symbol-file with the path of the real executable followed by the address (base + offset) retrieved using /proc/pid/maps (first result) and readelf.
but none worked...
The add-symbol-file ... solution should work. I suspect you are not supplying correct .text address.
cat /proc/30622/maps | grep "r-xp" | grep "/root/test/test"
This assumes that the very first segment of /root/test/test has RX permissions.
That used to be the case, but no longer is on modern systems (see e.g. this answer).
You didn't provide output from readelf -Wl /root/test/test, but I bet it looks similar to the 4-segment example from the other answer (with the first LOAD segment having Read only permissions.
Generally you need to find the address of the first LOAD segment of the test executable in memory, and add the address of .text to that base address.
Update:
With the newly-supplied output from /proc/$pid/maps and readelf, we can see that my guess was correct: this binary has 4 LOAD segments, and the first one doesn't have r-x permissions.
The calculation then is: $address_of_the_first_PT_LOAD + $address_of_.text. That is (for the 16873 process):
(gdb) add-symbol-file /root/test/test 0x7f5e37862000+0x10c0
objdump -s --section=".text" /root/test/test | grep Contents -A 1 ...
This is unnecessarily convoluted. Much easier way:
readelf -WS /root/test/test | grep '\.text ' | nawk '{print "0x" $4}'
Update:
I need to run gdb in the test machine (which doesn't have nawk and other bells
Part of the difficulty here is that you have a PIE binary (relocated at runtime).
If the problem reproduces for non-PIE binary (built with -fno-pie -no-pie), then the address can be calculated once (e.g. on the development machine) and reused on the test machine. You wouldn't need a script to compute the address every time you run the binary.
If I compile the binary to have the interpreter path hardcoded
Generally that's the best approach. Why not use it?
If you want the binary to run both on the development and the test machine, make copies of /root/test available on both.