I was trying to implement a stack tracer, using stack pointers; RSP and RBP, but I think debuggers use an entirely different way to grab the return addresses, or maybe I am missing something. I can grab the return address of the last stack frame, but I can't get the others because I don't know the size of other stack frames, so I can't figure out how much bytes should I go back from stack frame, to get the return address. Are there anybody know which way do debuggers use to trace stack?
It is possible to trace the stack when the code uses frame pointers. In this case ebp/rbp is used as the frame pointer and functions begin with prologs and end with epilogs.
A typical prolog looks like this:
push rbp ; save previous frame pointer
mov rbp, rsp ; initialize this functions frame pointer
A typical epilog looks like this:
mov rsp, rbp ; restore the value of rsp
pop rbp ; restore previous frame pointer value from stack
retn
Thus in every place in a function rbp points to the stack position where the previous frame pointer is saved and rbp+8 contains the saved return address.
To get the called function a debugger should read [rbp+8] value and find a function to which this address belongs. This can be done by searching in debugging symbols.
Next it should read [rbp] value to get the frame pointer of the caller function. Continue this process until you find a toplevel function. This is typically a system library function that starts threads.