Linux 64-abi, calling convention

Question

I'm reading intel manual about calling convention and which register has which purpose. Here is what was specified in the Figure 3.4: Register Usage:

%rax       temporary register; with variable arguments
           passes information about the number of vector
           registers used; 1st return register

But in linux api we use rax to pass the function number. Is it consistent with what was specified in the intel manual? Actually I expected that (according to the manual) we will pass the function number into rdi (it is used for the 1st argument). And so forth...

Can I use rax to pass the first function argument in my hand-written functions? E.g.

mov rax, [array_lenght_ptr]
mov rdi, array_start_ptr
callq _array_sum

Answer 1

That quote is talking about the function-calling convention, which is standardized by x86-64 System V ABI doc.

You're thinking of Linux's system-call calling convention, which is described in an appendix to the ABI doc, but that part isn't normative. Anyway, the system call ABI puts the call number in rax because it's not an arg to the system call. Alternatively, you can think of it like a 0th arg, the same way that variadic function calls pass the number of FP register-args in al. (Fun fact: that makes it possible for the caller to pass even the first FP arg on the stack if they want to.)

But more importantly, because call number in RAX makes a better ABI, and because of tradition: it's what the i386 system-call ABI does, too. And the i386 System V function-call ABI is totally different, using stack args exclusively.

This means system-call wrapper functions can just set eax and run syscall instead of needing to do something like

libc_write_wrapper_for_your_imagined_syscall_convention:
   ; copy all args to the next slot over
    mov r10, rdx   ; size_t count
    mov rdx, rsi   ; void *buf
    mov esi, edi   ; int fd
    mov edi, 1     ; SYS_write
    syscall
    cmp  rax, -4095
    jae  set_errno
    ret

instead of

actual_libc_write_wrapper:   ; glibc's actual code I think also checks for pthread cancellation points or something...
    mov eax, 1     ; SYS_write
    syscall
    cmp  rax, -4095
    jae  set_errno
    ret

Note the use of r10 instead of rcx because syscall clobbers rcx and r11 with the saved RIP and RFLAGS, so it doesn't have to write any memory with return information, and doesn't force user-space to put it somewhere the kernel can read it (like 32-bit sysenter does).

So the system-calling convention couldn't be the same as the function-calling convention. (Or the function-calling convention would have had to choose different registers.)

For system calls with 4 or more args (or a generic wrapper that works for any system call) you do need a mov r10, rcx, but that's all. (Unlike the 32-bit convention where a wrapper has to load args from the stack, and save/restore ebx because the kernel's poorly-chosen ABI uses it for the first arg.)

---

Can I use rax to pass the first function argument in my hand-written functions?

Yes, do whatever you want for private helper functions that you don't need to call from C.

Choose arg registers to make things easier for the callers (or for the most important caller), or you'll be using any registers with fixed register choices (like div).

Note which registers are clobbered and which are preserved with a comment. Only bother to save/restore registers that your caller actually needs saving / restoring, and choose which tmp regs you use to minimize push/pop. Avoid a push/pop save/reload of registers that are part of a critical latency path in your caller, if your function is short.