Search code examples
dockergdbserver

gdbserver does not attach to a running process in a docker container

In my docker container (based on SUSE distribution SLES 15) both the C++ executable (with debug enhanced code) and the gdbserver executable are installed.
Before doing anything productive the C++ executable sleeps for 5 seconds, then initializes and processes data from a database. The processing time is long enough to attach it to gdbserver.
The C++ executable is started in the background and its process id is returned to the console.
Immediately afterwards the gdbserver is started and attaches to the same process id.

Problem: The gdbserver complains not being able to connect to the process:

Cannot attach to lwp 59: No such file or directory (2)
Exiting

In another attempt, I have copied the same gdbserver executable to /tmp in the docker container.
Starting this gdbserver gave a different error response:

Cannot attach to process 220: Operation not permitted (1)
Exiting

It has been verified, that in both cases the process is still running. 'ps -e' clearly shows the process id and the process name.
If the process is already finished, a different error message is thrown; this is clear and needs not be explained:

gdbserver: unable to open /proc file '/proc/79/status'

The gdbserver was started once from outside of the container and once from inside.
In both scenarios the gdbserver refused to attach the running process:

  1. $ kubectl exec -it POD_NAME --container debugger -- gdbserver --attach :44444 59
    Cannot attach to lwp 59: No such file or directory (2)
    Exiting
  2. $ kubectl exec -it POD_NAME -- /bin/bash
    bash-4.4$ cd /tmp
    bash-4.4$ ./gdbserver 10.0.2.15:44444 --attach 220
    Cannot attach to process 220: Operation not permitted (1)
    Exiting

Can someone explain what causes gdbserver refusing to attach to the specified process and give advice how to overcome the mismatch, i.e. where/what do I need to examine for to prepare the right handshake between the C++ executable and the gdbserver?

Solution

  • The basic reason why gdbserver could not attach to the running C++ process is due to a security enhancement in Ubuntu (versions >= 10.10):
    By default, process A cannot trace a running process B unless B is a direct child of A (or A runs as root).
    Direct debugging is still always allowed, e.g. gdb EXE and strace EXE.

    The restriction can be loosen by changing the value of /proc/sys/kernel/yama/ptrace_scope from 1 (=default) to 0 (=tracing allowed for all processes). The security setting can be changed with:
    echo 0 | sudo tee /proc/sys/kernel/yama/ptrace_scope

    All credits for the description of ptrace scope belong to the following post, see 2nd answer by Eliah Kagan - thank you for the thorough explanation! - here:
    https://askubuntu.com/questions/143561/why-wont-strace-gdb-attach-to-a-process-even-though-im-root