I'm trying to try out writing to the console in assembly. I ended up using Windows API WriteFile function. It works well, but I have encountered a weird quirk. If I don't have the rbx register set to [rsp], the WriteFile doesn't return and gives the error code 0xC0000005, which is an access violation error. It still writes correctly to the console, which I find weird. It isn't a problem and I can include one line, no problem. But it bothers me, because I cant figure out why this behaviour exists.
If it does help, I'm using NASM and GCC to compile my program.
Here is the entire code for context:
bits 64
default rel
extern GetStdHandle
extern WriteFile
extern ExitProcess
section .data
buffer db 0
section .text
print:
push rbp
mov rbp, rsp
add rsp, 8
mov rcx, -11
call GetStdHandle
mov rcx, rax
mov rax, 0x61616161
mov [buffer], rax
mov rdx, buffer
mov r8, 4
mov rbx, [rsp]
call WriteFile
mov rsp, rbp
pop rbp
ret
global main
main:
call print
mov rcx, 0
call ExitProcess
I tried moving the content of the function to replace call print and in that case it works even without mov rbx, [rsp]. I also tried setting the rbx register to other values, but [rsp] seems to be the only one that works.
The Windows calling convention requires the caller to provide 32 bytes of free space on the stack above the return address for the called function to use. Because this space is not provided, the WriteFile function overwrites parts of the caller's stack. In particular, it is overwriting the return address with the value in rbx. Loading rbx with the return address masks this bug.
To fix it, change add rsp, 8 to sub rsp, 32. (And of course remove the instruction that changes rbx. The function must not change rbx without preserving its value.)