I am trying to debug a sigsegv on a service startup using gdb. I have configured debuginfod and it is even downloading debuginfo but still it is showing ?? instead of symbols. Here is the relevant output:
GNU gdb (GDB) 15.1
Copyright (C) 2024 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./build/dns-wrapper...
(gdb) r
Starting program: /home/ncd/git/dns-wrapper/build/dns-wrapper --config-file config/config.ini
Downloading separate debug info for /lib64/ld-linux-x86-64.so.2
Downloading separate debug info for system-supplied DSO at 0x7ffff7fc5000
Downloading separate debug info for /usr/lib/libboost_system.so.1.86.0
Downloading separate debug info for /usr/lib/libboost_program_options.so.1.86.0
Downloading separate debug info for /usr/lib/libboost_log_setup.so.1.86.0
Downloading separate debug info for /usr/lib/libboost_log.so.1.86.0
Downloading separate debug info for /usr/lib/libboost_thread.so.1.86.0
Downloading separate debug info for /usr/lib/libboost_atomic.so.1.86.0
Downloading separate debug info for /usr/lib/libboost_chrono.so.1.86.0
Downloading separate debug info for /usr/lib/libboost_filesystem.so.1.86.0
Downloading separate debug info for /usr/lib/libm.so.6
Downloading separate debug info for /usr/lib/libc.so.6
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".
Upstream server identified as: udp://1.1.1.1:53
Upstream server identified as: udp://8.8.8.8:53
[2024-09-28 18:53:39.662405] [0x00007ffff7d39940] [trace] [0x0008a1ea] [1] Dns Wrapper v0.0.0.1
[2024-09-28 18:53:39.662521] [0x00007ffff7d39940] [trace] [0x0008a1ea] [2] Git branch: win_raw_socket, Version: 01b16b2e-dirty, Commit: 01b16b2e-dirty, Date: 2024-09-28 10:24:37 +0530
[2024-09-28 18:53:39.662533] [0x00007ffff7d39940] [trace] [0x0008a1ea] [3] Current User: root
[2024-09-28 18:53:39.662540] [0x00007ffff7d39940] [trace] [0x0008a1ea] [4] Compiled for x86_64 (compiled locally) using cc (GCC) 14.2.1 20240910
[2024-09-28 18:53:39.662760] [0x00007ffff7d39940] [info] [0x0008a1ea] [5] Successfully read rules from file: ./rules.txt
[2024-09-28 18:53:39.662997] [0x00007ffff7d39940] [debug] [0x0008a1ea] [6] Skipping entering into daemon mode as per user request
[2024-09-28 18:53:39.663053] [0x00007ffff7d39940] [trace] [0x0008a1ea] [7] Daemon started on port: 10053
Program received signal SIGSEGV, Segmentation fault.
0x00005555555f43ed in boost::asio::detail::socket_ops::call_recvfrom<unsigned int> (s=8, data=0x7ffffffeab80, size=65535, flags=0, addr=0x7fffffffa740,
addrlen=0x7fffffffa890) at /usr/include/boost/asio/detail/impl/socket_ops.ipp:1130
1130 }
(gdb) where
#0 0x00005555555f43ed in boost::asio::detail::socket_ops::call_recvfrom<unsigned int> (s=8, data=0x7ffffffeab80, size=65535, flags=0, addr=0x7fffffffa740,
addrlen=0x7fffffffa890) at /usr/include/boost/asio/detail/impl/socket_ops.ipp:1130
#1 0x2851232006040001 in ?? ()
#2 0x00007fff0000fe51 in ?? ()
#3 0x0000000000000000 in ?? ()
(gdb) show debuginfod enabled
Debuginfod functionality is currently set to "on".
(gdb) show debuginfod urls
Debuginfod URLs are currently set to:
https://debuginfod.archlinux.org
(gdb) bt full
#0 0x00005555555f43ed in boost::asio::detail::socket_ops::call_recvfrom<unsigned int> (s=8, data=0x7ffffffeab80, size=65535, flags=0, addr=0x7fffffffa740,
addrlen=0x7fffffffa890) at /usr/include/boost/asio/detail/impl/socket_ops.ipp:1130
tmp_addrlen = 20
result = 561
#1 0x2851232006040001 in ?? ()
No symbol table info available.
#2 0x00007fff0000fe51 in ?? ()
No symbol table info available.
#3 0x0000000000000000 in ?? ()
No symbol table info available.
As you can see gdb downloaded symbols and there were no errors. However I still see ?? () in stack trace. Is there something I am missing from my side? How can I get gdb to show the symbols?
I am running gdb like so: sudo DEBUGINFOD_URLS=https://debuginfod.archlinux.org gdb --args ./build/dns-wrapper --config-file config/config.ini
Running it as root since service requires root privileges.
#0 0x00005555555f43ed in boost::asio::detail::socket_ops::call_recvfrom<unsigned int> (s=8, data=0x7ffffffeab80, size=65535, flags=0, addr=0x7fffffffa740, addrlen=0x7fffffffa890) at /usr/include/boost/asio/detail/impl/socket_ops.ipp:1130
#1 0x2851232006040001 in ?? ()
This typically means that stack overflow (in call_recvfrom() or something it called) caused the return address on the stack to be overwritten.
Any suggestions how to debug this further?
Use -fsanitize=address to build this program with address sanitizer -- it would likely point you straight at the bug.
Also note that 0x28512320 06040001 spells "(Q# " in ASCII. If these are possible in the packet you are receiving, that's another confirmation of stack overflow.
P.S. Valgrind (suggested by OznOg) will likely not detect anything (its detection of stack overflows is exceedingly weak).