Search code examples
assemblyx86-64stack-overflowinline-assembly

Retq instruction, where does it return


I am unable to understand where the assembly instruction retq returns to.

I understand that when my normal code executes then it return to the address specified in the stack. But how does it know where in the stack is the return address located?

In short, does it use rbp or esp for finding the address on the stack?


Solution

  • After studying assembly code, here are my thoughts, let's look at a sample:

    fun:
    push %rbp
    mov %rsp,%rbp
    ...
    ...
    pop %rbp
    retq
    
    main:
    ...
    ...
    callq  "address" <fun>
    ...
    ...
    

    We can see there is a instruction before retq. The pop %rbp (sometimes it is a leave instruction but they are similar) instruction will

    1. save the content of current stack pointer %rsp to base stack pointer %rbp.
    2. move the %rsp pointer to previous address on stack.

    For example: before pop command, the %rsp pointed to 0x0000 0000 0000 00D0. After the pop command it points to 0x0000 0000 0000 00D8 (assume the stack grows from high address to low address).

    After the pop command, now %rsp points to a new address and retq takes this address as return address.