Remote Setup
I need to debug a complex C++ program which is installed in a docker container, controlled by Kubernetes. The docker container also provides a gdbserver and exposes the container port 44444.
Host Setup
The gdb part to control and examine the program is set in a another docker container. This is due to the case, that the SUSE environment is only available in this container, not on my Ubuntu 18.04 machine in a VM Box.
Local debugging works well
Debugging the program locally in the SUSE docker container works well. The program halts at the specified breakpoints and these breakpoints are also specified in remote debugging. All breakpoints are solely defined in the program's basic source code files, not in any libs.
It has been verified, that the executable in the remote docker container is identical to the one in the host container; it has been compiled with debug symbols and non-optimized code (-ggdb -O0).
Problem
Debugging the program remotely only lacks in stopping at the defined breakpoints on the host. The program in the container is started in background. When gdbserver attaches its process_id the program halts until 'continue' is issued within the gdb host session and forwarded to gdbserver in the remote container.
The program is deployed with basic C++ class files and shared program libraries together with shared project libraries. It is started with parameters and exits after the job is done.
When the program is started it reads configuration files, connects to a database, reads database entries, prepares and formats the data to XML formated entries and writes them into an output file.
HelloWorld remote debugging test works well
To verify that the remote debugging setup and connection via gdbserver port works well, I created a simple HelloWorld C++ program and copied this into the same remote docker container and tested the breakpoint behaviour therein.
The remote debug test scenario is working successfully when the HelloWorld program is run in the container:
the internal container port 44444 is mapped to the same external port id 44444:
$ kubectl port-forward eric-bss-eb-tools-65c4955565-xdqtx 44444:44444
Forwarding from 127.0.0.1:44444 -> 44444
Forwarding from [::1]:44444 -> 44444
HelloWorld in the remote container is started in background and sleeps a few seconds
bash-4.4$ ./HelloWorld &
[1] 1068
gdbserver attaches to the HelloWorld process_id and waits to forward the gdb commands
bash-4.4$ ./gdbserver :44444 --attach 1068 // gdbserver uses the exposed port
Attached; pid = 1068
Listening on port 44444
gdb in the host container is started in the HelloWorld source code folder with TUI mode
$ gdb -tui HelloWorld
reading symbols from HelloWorld...done.
(gdb) b 13
Breakpoint 1 at 0x400b2d: file HelloWorld.cpp, line 13.
(gdb) b 15
Breakpoint 2 at 0x400b37: file HelloWorld.cpp, line 15.
gdb connects to the gdbserver via localhost and (external) port id 44444
(gdb) target remote :44444
(gdb) c
Continuing.
the remote HelloWorld stops at breakpoint 2; variables can be examined; further gdb commands like 'next' and 'step' can be issued; everything works smart
Target program remote debugging doesn't stop at breakpoints
When the target C++ program in the container is debugged with the same scenario it does not stop at the defined breakpoints:
the workflow is identical to the HelloWorld test scenario with the exception,
that the breakpoints are defined after gdb has made the connection to gdbserver
(target remote :44444).
This has been done as per the advice in the 2nd comment from this answer: (Remote gdb debugging does not stop at breakpoints).
Nevertheless, the breakpoints are still ignored even when they are defined after the connection to the remote target has been established.
the program in the remote docker container is halted by the gdbserver and continues its execution when gdb issues the 'continue' command, but does not stop at any of the breakpoints.
I tried several hints as per other analogous problem descriptions, but breakpoints are still ignored.
E.g. using hardware break points as been advised in the answer of the same request here:
(Remote gdb debugging does not stop at breakpoints)
Running the remote docker container with securityContext: privileged=true is forbidden in my environment, hence this could not be tested. See proposal here: (gdb does not hit any breakpoints when I run it from inside Docker container)
What am I missing to get remote debugging in a docker container halted at defined breakpoints?
Due to a security enhancement in Ubuntu (versions >= 10.10), users are not allowed to ptrace processes that are not a descendant of the debugger.
By default, process A cannot trace a running process B unless B is a direct child of A
(or A runs as root).
Direct debugging is still always allowed, e.g. gdb EXE and strace EXE.
The restriction can be loosen by changing the value of /proc/sys/kernel/yama/ptrace_scope from 1 (=default) to 0 (=tracing allowed for all processes). The security setting can be changed with:
echo 0 | sudo tee /proc/sys/kernel/yama/ptrace_scope
HelloWorld remote debugging test works well
How did it happen, that remote debugging in the HelloWorld container worked well?
The HelloWorld container was created with USER userName in the Dockerfile
which is the same user name as been logged in to Ubuntu.
The Dockerfile for to deploy the development container (with the C++ program to be debugged) defines both a different user name and group name than being used in my Ubuntu login.
All credits for the description of ptrace scope belong to the following post,
see 2nd answer by Eliah Kagan - thank you for the thorough explanation! - here:
https://askubuntu.com/questions/143561/why-wont-strace-gdb-attach-to-a-process-even-though-im-root