I have an C-application which runs on many machines and from time to time an instance makes issues and behaves weird. Unfortunately, this happens almost never. These prod-instances are compiled with heavy optimizations (-march=XXX -Ofast) and do not include any debug-symbols, so I cannot easily attach a debugger to analyze their state.
But I thought it should be possible to compile the application again with the same flags plus -g3 and then I can load it in gdb with symbol-file application_executable_with_debug_symbols. However, if I do that then breakpoints never trigger.
Is there another way to attach a debugger to a running application and loading debug-symbols? Or is there something (obvious) which I do wrong?
Thanks
The best practice is to build the application with debug symbols, keep the resulting binary for debugging, but run strip -g app.debug -o app.release and run the stripped binary in production.
When you find a misbehaving instance, you can copy the full-debug version to target machine, and run gdb -ex 'attach $PID' app.debug. Voila: you have full debug symbols.
The most likely reason that compiling the application again didn't work is that you got a new binary with different symbols (compare nm app.debug vs. nm app.release), and the most likely reason for that (if using GCC) is that you omitted some of the optimization flags used to build app.release, or you used slightly different sources -- you must use exactly the same flags (and add -g) and exactly the same sources for any hope of success with that approach.