I tried to use gdb to analyze the coredump file. The stack frame was damaged so the bt instruction could not be used directly. I tried to use rbp to backtrace manually. The rbp of my first stack frame is 0x7fxxxxxxxxxx, so I suspect it is in kernel mode.
How should I return to the user stack?
so I suspect it is in kernel mode.
Your suspicion is wrong:
RSP
(and RBP
) for the main thread of an x86_64
program typically looks like this: 0x7fffffffd8b0
-- there is nothing "kernely" about it.