Search code examples
cassemblystackcall

Calling convention and stack, returning value from call?

Does anyone know what happens after a call pushes a value on the stack, it is popped off by callee, then the return address is popped off by callee? Perhaps with some assembly?

I think for the following code the return value of fib is then used as an argument passed to plus function. What I’m confused about is fib return value is popped into a register, then somehow that value ends up being pushed on to a stack frame for the next call.

int fib(int n) {
   if (n == 0)
      return 0;
   if (n == 1)
      return 1;
   return fib(n - 1) + fib(n - 2);
}

int main() {
   return fib(3);
}

I’m using the following picture to understand a thread safe way to use the stack and understand that the return value may be in a register for some conventions.

ea Ref: link

Solution

  • There is a number of calling conventions in C, most notably cdecl and stdcall.

    Under cdecl, the caller is responsible for balancing the stack.

    This is done by following every call instruction (to a function accepting at least one parameter) with a stack-balancing instruction that pops arguments from the stack once the function returns.

    In x86, this is done with one or two pop instructions, or an add sp, N instruction.

    In LC3, this is done with an ADD R6, R6, N instruction.

    Under stdcall, it is the callee who is responsible for balancing the stack.

    This is done by popping arguments from the stack right before the function returns.

    In x86, this is done with a special ret N instruction at the end of the function, which pops the return address from the stack, then pops and discards N bytes from the stack, and then jumps to the return address.

    In LC3, either there is no suitable instruction, or I was unable to find such an instruction. So maybe LC3 does not use stdcall.

    x86 assembly provided by Peter Cordes in a comment: https://godbolt.org/z/jb31fhPza

    Since you have not specified a "calling convention" for fib(), your compiler uses the default calling convention, which, if you are doing x86, is probably cdecl, but you better check with your compiler.