The following assembly code causes a segmentation fault exiting from the main function at the pop rbp instruction. This code was generated by a compiler I'm writing, so don't mind the superfluous instructions:
.intel_syntax noprefix
.global main
.text
add:
push rbp
mov rbp, rsp
sub rsp, 4
mov dword [rbp - 4], edi
sub rsp, 4
mov dword [rbp - 8], esi
mov eax, dword [rbp - 4]
mov ebx, dword [rbp - 8]
add eax, ebx
mov rsp, rbp
pop rbp
ret
main:
push rbp
mov rbp, rsp
mov eax, 60
mov edi, eax
mov eax, 9
mov esi, eax
call add
mov rsp, rbp
pop rbp
ret
add rsp, 4
I've double checked I'm keeping the stack in order, so I don't see how this error could be occurring.
I've tried debugging it with GDB, but with not much success.
You want dword ptr.
Just plain dword is defined as 4 like in MASM, so mov dword [rbp - 4], edi is actually
mov 4[rbp - 4], edi = mov [rbp + 0], edi, overwriting the saved RBP. (So your add function corrupts the caller's RBP).
In GDB with a "disassembly" view while single-stepping (layout asm), this was immediately obvious; I was already suspicious of that code because of those inefficient sub rsp, 4 / mov pairs - you should just change RSP once on function entry with sub rsp, 8 to reserve all the stack space you're going to need, ahead of both stores.
(Or perhaps use push rdi / mov [rbp-4], esi, but if you ever plan to do register allocation for variables instead of always spilling them on function entry, most functions won't need that optimization; mainstream C compilers don't look for it even in debug builds where they do always spill everything.)
For debugging stuff like this, watch -l *(long*)$rbp inside add (after push rbp / mov rbp, rsp) also works: continue from there shows the mov DWORD PTR [rbp+0x0],edi changing that memory location.
The other clue is that RSP = 0x7fff0000003c when pop rbp faults, which should make you suspect that the low half has been overwritten with a small integer. (And since we just ran mov rsp, rbp, and the RBP value is similarly corrupted, that's the obvious thing to suspect.)
Also, you wouldn't have this problem with AT&T syntax, or with NASM where keywords like dword aren't land-mines that will silently break your code if you use them wrong.
GAS .intel_syntax is not as robust. For example GCC itself will make broken asm if you have a global variable called long rax; or int offset;. A variable named "offset" causes "Error: invalid use of register" to appear when using "-masm=intel" in gcc, but no error in AT&T mode.
It's possible to call "rax" with the quotes escaping it from being treated as a register name, but GCC doesn't do that; -masm=intel is a second-class citizen for GCC, intended mostly for humans to read the asm, apparently not for production use. (Disambiguate labels from register names in the Intel syntax) In NASM, you'd need call $rax to force treating it as a symbol.
GAS .intel_syntax also has an ambiguity depending on .equ appearing before or after code that uses it. Distinguishing memory from constant in GNU as .intel_syntax
I always prefer Intel syntax for disassembly, but most projects with hand-written asm don't use it for their source code. For automated code-gen, AT&T is probably a better bet, unless you want to support inline asm using Intel syntax. Then just be aware of the limitations.