Per https://www.gnu.org/software/libc/manual/html_node/Backtraces.html:
Note that certain compiler optimizations may interfere with obtaining a valid backtrace. Function inlining causes the inlined function to not have a stack frame; tail call optimization replaces one stack frame with another; frame pointer elimination will stop backtrace from interpreting the stack contents correctly.
I want to know why it should happen when enabling tail optimization and how to avoid it.
I want to know why it should happen when enabling tail optimization
It will happen with tail call optimization because your recursive function will not be recursive after the compiler is done with it, or a tail recursive function will be entirely missing from the call stack.
Example:
// t.c
int foo() { return 42; }
int bar() { return foo(); }
int main() { return bar(); }
gcc -O2 -fno-inline t.c && gdb -q ./a.out
(gdb) b foo
Breakpoint 1 at 0x1140
(gdb) run
Starting program: /tmp/a.out
Breakpoint 1, 0x0000555555555140 in foo ()
(gdb) bt
#0 0x0000555555555140 in foo ()
#1 0x00007ffff7dfad0a in __libc_start_main (main=0x555555555040 <main>, argc=1, argv=0x7fffffffdc08, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffdbf8) at ../csu/libc-start.c:308
#2 0x000055555555507a in _start ()
Where did main() and bar() go? We know that main() must still be on the stack.
They were tail call optimized:
(gdb) disas main
Dump of assembler code for function main:
0x0000555555555040 <+0>: xor %eax,%eax
0x0000555555555042 <+2>: jmpq 0x555555555150 <bar>
End of assembler dump.
(gdb) disas bar
Dump of assembler code for function bar:
0x0000555555555150 <+0>: xor %eax,%eax
0x0000555555555152 <+2>: jmp 0x555555555140 <foo>
End of assembler dump.
how to avoid it.
The only way to avoid it is to disable tail call optimization (with -fno-optimize-sibling-calls):
gcc -O2 -fno-inline t.c -fno-optimize-sibling-calls && gdb -q ./a.out
(gdb) b foo
Breakpoint 1 at 0x1140
(gdb) run
Starting program: /tmp/a.out
Breakpoint 1, 0x0000555555555140 in foo ()
(gdb) bt
#0 0x0000555555555140 in foo ()
#1 0x0000555555555157 in bar ()
#2 0x0000555555555047 in main ()