My assembly skills are rather poor, but I've been trying to understand assembly a little better to improve my understanding of profiling sessions and how optimizing compilers work.
One of the things that I never stopped to really think about at the machine level when studying 64-bit calling conventions for Windows is that given something like this:
struct BigUdt
{
double buf[16]; // exceeds 64 bits
};
struct BigUdt create_big_udt();
... if I understood the documentation properly, create_big_udt would normally allocate the memory for struct BigUdt (I'm assuming on the stack) and return the address to its memory in rax.
So my question, from an assembly standpoint, is this: where does that leave the caller as far as responsibilities go? I'm assuming someone is left with the responsibility of adding sizeof(BidUdt) back to rsp at some point. Also would 64-bit stack alignment become a concern here for types whose sizes aren't a multiple of 64-bits, or would that be covered by struct BigUdt's padding?
Phew, I hope that's a reasonable question. I find these ABI contracts between caller and callee with calling conventions quite intimidating and one of the hardest parts of learning assembly so far.
The documentation is quite clear:
Otherwise, the caller assumes the responsibility of allocating memory and passing a pointer for the return value as the first argument. Subsequent arguments are then shifted one argument to the right. The same pointer must be returned by the callee in RAX.
So basically foo bar(...) gets turned into foo* bar(foo* retval, ...). The pointer may point to wherever the caller wants the result (typically it will be the address of a local variable in the caller's stack frame) which is of course unaffected by the stack operations involved in the call, and is always valid memory, stack or otherwise. It's never "hanging in the wind" (if I understood the term properly), since that would indeed be a recipe for disaster.