Search code examples
assemblymipscalling-conventionstack-framestack-pointer

Does the prologue of a function can write outside of its frame?

I'm currently trying to analyse the assembly of an old video game from N64. In order to do so, I'm using some N64 debugger to read and understand the underlying MIPS code.

In one of the calls I'm looking at, the prologue is defined as follows:

ADDIR SP, SP, -0x030
SW    RA, 0x0024 (SP)
SW    S0, 0x0020 (SP)
SW    A1, 0x0034 (SP)

Later on this function, we also have two more stack pushes:

SW    V0, 0x002C (SP)
[...]
SW    R0, 0x0010 (SP)

What I don't understand is:

  • why the prologue subs 0x030, but doesn't use all the space: we store only 5 registers, so it should subs at most 0x014
  • why, and this is the main issue for me here, A1 is store outside of SP? The stack subs 0x030, but A1 is save to SP[0x034]

I'm no expert at all on ASM, so I may miss something on how stack works, but for me every saved data in the prologue should be saved beetwen SP ans SP - 0x030 (in my case). If I understand correctly, the fourth line writes on the stack frame of another function, which seems bad.

Solution

  • One MIPS calling convention has the caller allocate stack memory space for all the parameters despite that the first 4 parameters (at least) are actually passed in registers.

    This means that the called function can store $a0, $a1, $a2, $a3 to the stack, expecting that those memory locations are available.

    When a function calls another function, then as a caller it should also allocate those same 4 words of stack space.  A function that calls another will need at least a minimal stack frame anyway, so having it allocate those 4 extra words is free in terms of prologue and epilogue.

    Part of older calling conventions would tend to include support for varargs (variadic functions) that may not even be properly declared in C code as such.  Allowing functions to store their parameters back to caller-allocated memory allows all the arguments to be flushed to memory and be contiguous, which is important for varargs functions.  And doing it for all functions is overkill but simplifies some issues.

    See https://courses.cs.washington.edu/courses/cse410/09sp/examples/MIPSCallingConventionsSummary.pdf for a picture of the stack frame that includes the 4 words for the register arguments.

    In practice I believe that the reservation of 4 words for the callee to use in the caller's stack space is highly overrated, and as evidence would note that this has been dropped in RISC V, for example.

    Some Intel systems use a red zone instead, which says that the system agrees not use your stack space within a certain small distance below the stack pointer (i.e. in the unallocated space of the stack).  This makes sense on intel as the return address is automatically written to memory so doesn't need to be separately pushed like on MIPS, yet these systems also benefit from some pre-allocated stack space for simple functions to use without necessarily setting up a stack frame (i.e. the red zone).