Search code examples
winapiassembly64-bitx86-64nasm

Repeated call of WriteConsole (NASM x64 on Win64)


I started to learn Assembly lately and for practice, I thought of makeing a small game. To make the border graphic of the game I need to print a block character n times. To test this, I wrote the following code:

bits 64

global main
extern ExitProcess
extern GetStdHandle
extern WriteConsoleA

section .text
    
main:
    mov rcx, -11
    call GetStdHandle   
    mov rbx, rax
drawFrame:
    mov r12, [sze]
    l:
    mov rcx, rbx
    mov rdx, msg
    mov r8, 1
    sub rsp, 48
    mov r9, [rsp+40]
    mov qword [rsp+32], 0
    call WriteConsoleA
    dec r12
    jnz l
    
    xor rcx, rcx
    call ExitProcess

section .data
    score dd 0
    sze dq 20
    msg db 0xdb

I wanted to make this with the WinAPI Function for ouput. Interestingly, this code stops after printing one char when using WriteConsoleA, but when I use C's putchar, it works correctly. I could also manage to make a C equivalent with the WriteConsoleA function, which also works fine. The disassembly of the C code didn't bring me further.

I suspect there's something wrong in my use of the stack that I don't see. Hopefully someone can explain or point out.


Solution

  • You don't want to keep subtracting 48 from RSP through each loop. You only need to allocate that space once before the loop and before you call a C library function or the WinAPI.

    The primary problem is with your 4th parameter in R9. The WriteConsole function is defined as:

    BOOL WINAPI WriteConsole(
      _In_             HANDLE  hConsoleOutput,
      _In_       const VOID    *lpBuffer,
      _In_             DWORD   nNumberOfCharsToWrite,
      _Out_opt_        LPDWORD lpNumberOfCharsWritten,
      _Reserved_       LPVOID  lpReserved
    );
    

    R9 is supposed to be a pointer to a memory location that returns a DWORD with the number of characters written, but you do:

    mov r9, [rsp+40]
    

    This moves the 8 bytes starting at memory address RSP+40 to R9. What you want is the address of [rsp+40] which can be done using the LEA instruction:

    lea r9, [rsp+40]
    

    Your code could have looked like:

    bits 64
    
    global main
    extern ExitProcess
    extern GetStdHandle
    extern WriteConsoleA
    
    section .text
        
    main:
        sub rsp, 56          ; Allocate space for local variable(s)
                             ; Allocate 32 bytes of space for shadow store
                             ; Maintain 16 byte stack alignment for WinAPI/C library calls
                             ; 56+8=64 . 64 is evenly divisible by 16.
        mov rcx, -11
        call GetStdHandle   
        mov rbx, rax
    drawFrame:
        mov r12, [sze]
    l:
        mov rcx, rbx
        mov rdx, msg
        mov r8, 1
        lea r9, [rsp+40]
        mov qword [rsp+32], 0
        call WriteConsoleA
        dec r12
        jnz l
        
        xor rcx, rcx
        call ExitProcess
    
    section .data
        score dd 0
        sze dq 20
        msg db 0xdb
    

    Important Note: In order to be compliant with the 64-bit Microsoft ABI you must maintain the 16 byte alignment of the stack pointer prior to calling a WinAPI or C library function. Upon calling the main function the stack pointer (RSP) was 16 byte aligned. At the point the main function starts executing the stack is misaligned by 8 because the 8 byte return address was pushed on the stack. 48+8=56 doesn't get you back on a 16 byte aligned stack address (56 is not evenly divisible by 16) but 56+8=64 does. 64 is evenly divisible by 16.