Search code examples
visual-studioassemblyvisual-studio-2012masm

Why does the program assemble, but do nothing?


Here is a copy of the source:

extrn MessageBoxA: PROC
extrn ExitProcess: PROC

.data
mytit db 'The 64-bit world of Windows & assembler...', 0
mymsg db 'Hello World!', 0

.code
main proc
mov r9d, 0       ; uType = MB_OK
lea r8,  mytit   ; LPCSTR lpCaption
lea rdx, mymsg   ; LPCSTR lpText
mov rcx, 0       ; hWnd = HWND_DESKTOP
call MessageBoxA
mov ecx, eax     ; uExitCode = MessageBox(...)
call ExitProcess
main endp

End

Right now, I am just trying to get my 'first' x64 assembly program up and running so that I can start playing around and actually learn some assembly, so I just literally copied this source from here in an attempt to see if I could get anything to assemble correctly, but so far, no luck.

If I assemble this, I don't get any errors, neither during assembly-time nor run-time, but the program doesn't run as it's supposed to: it doesn't appear to do anything at all. Once the executable has been generated and I double-click on it, nothing happens, and if I go to the task manager, it doesn't seem to be running in the background either. What's going on?

I am using the default settings for MASM64 that are generated upon checking "masm" under "Build Customizations..." (found by right-clicking the project in the Solution Explorer) and changing the platform from Win32 to x64 in the Configuation Manager, the exceptions being that I've changed the "Entry Point" linker option to "main" and the "SubSystem" linker option to "Windows". (All of this is done in Visual Studio 2012.)

The debug info generated by running the program in Visual Studio is

'Hello World (ASM).exe' (Win32): Loaded 'D:\Google Drive\My Documents\Visual Studio 
2012\Projects\Hello World (ASM)\x64\Release\Hello World (ASM).exe'. Symbols loaded.
'Hello World (ASM).exe' (Win32): Loaded 'C:\Windows\System32\ntdll.dll'. Cannot find or open the PDB file.
'Hello World (ASM).exe' (Win32): Loaded 'C:\Windows\System32\kernel32.dll'. Cannot find or open the PDB file.
'Hello World (ASM).exe' (Win32): Loaded 'C:\Windows\System32\KernelBase.dll'. Cannot find or open the PDB file.
'Hello World (ASM).exe' (Win32): Loaded 'C:\Windows\System32\user32.dll'. Cannot find or open the PDB file.
'Hello World (ASM).exe' (Win32): Loaded 'C:\Windows\System32\gdi32.dll'. Cannot find or open the PDB file.
'Hello World (ASM).exe' (Win32): Loaded 'C:\Windows\System32\imm32.dll'. Cannot find or open the PDB file.
'Hello World (ASM).exe' (Win32): Loaded 'C:\Windows\System32\msctf.dll'. Cannot find or open the PDB file.
'Hello World (ASM).exe' (Win32): Loaded 'C:\Windows\System32\msvcrt.dll'. Cannot find or open the PDB file.
'Hello World (ASM).exe' (Win32): Loaded 'C:\Windows\System32\nvinitx.dll'. Cannot find or open the PDB file.
'Hello World (ASM).exe' (Win32): Loaded 'C:\Windows\System32\advapi32.dll'. Cannot find or open the PDB file.
'Hello World (ASM).exe' (Win32): Loaded 'C:\Windows\System32\sechost.dll'. Cannot find or open the PDB file.
'Hello World (ASM).exe' (Win32): Loaded 'C:\Windows\System32\rpcrt4.dll'. Cannot find or open the PDB file.
'Hello World (ASM).exe' (Win32): Loaded 'C:\Program Files\NVIDIA Corporation\coprocmanager\detoured.dll'. Cannot find or open the PDB file.
'Hello World (ASM).exe' (Win32): Loaded 'C:\Program Files\NVIDIA Corporation\coprocmanager\Nvd3d9wrapx.dll'. Cannot find or open the PDB file.
'Hello World (ASM).exe' (Win32): Loaded 'C:\Windows\System32\setupapi.dll'. Cannot find or open the PDB file.
'Hello World (ASM).exe' (Win32): Loaded 'C:\Windows\System32\cfgmgr32.dll'. Cannot find or open the PDB file.
'Hello World (ASM).exe' (Win32): Loaded 'C:\Windows\System32\devobj.dll'. Cannot find or open the PDB file.
'Hello World (ASM).exe' (Win32): Loaded 'C:\Program Files\NVIDIA Corporation\coprocmanager\nvdxgiwrapx.dll'. Cannot find or open the PDB file.
'Hello World (ASM).exe' (Win32): Loaded 'C:\PROGRA~1\NVIDIA~1\NVSTRE~1\rxinput.dll'. Module was built without symbols.
'Hello World (ASM).exe' (Win32): Loaded 'C:\Windows\System32\ole32.dll'. Cannot find or open the PDB file.
'Hello World (ASM).exe' (Win32): Loaded 'C:\Windows\System32\oleaut32.dll'. Cannot find or open the PDB file.
'Hello World (ASM).exe' (Win32): Loaded 'C:\Windows\System32\combase.dll'. Cannot find or open the PDB file.
'Hello World (ASM).exe' (Win32): Unloaded 'C:\PROGRA~1\NVIDIA~1\NVSTRE~1\rxinput.dll'
'Hello World (ASM).exe' (Win32): Unloaded 'C:\Windows\System32\combase.dll'
'Hello World (ASM).exe' (Win32): Unloaded 'C:\Windows\System32\oleaut32.dll'
'Hello World (ASM).exe' (Win32): Unloaded 'C:\Windows\System32\ole32.dll'
First-chance exception at 0x000007FA9A09B9FB (gdi32.dll) in Hello World (ASM).exe: 0xC0000005: Access violation reading location 0xFFFFFFFFFFFFFFFF.
Unhandled exception at 0x000007FA9A09B9FB (gdi32.dll) in Hello World (ASM).exe: 0xC0000005: Access violation reading location 0xFFFFFFFFFFFFFFFF.
First-chance exception at 0x000007FA9A09B9FB (gdi32.dll) in Hello World (ASM).exe: 0xC0000005: Access violation reading location 0xFFFFFFFFFFFFFFFF.
Unhandled exception at 0x000007FA9A09B9FB (gdi32.dll) in Hello World (ASM).exe: 0xC0000005: Access violation reading location 0xFFFFFFFFFFFFFFFF.
First-chance exception at 0x000007FA9A09B9FB (gdi32.dll) in Hello World (ASM).exe: 0xC0000005: Access violation reading location 0xFFFFFFFFFFFFFFFF.
Unhandled exception at 0x000007FA9A09B9FB (gdi32.dll) in Hello World (ASM).exe: 0xC0000005: Access violation reading location 0xFFFFFFFFFFFFFFFF.
The program '[1080] Hello World (ASM).exe' has exited with code 0 (0x0).

Solution

  • Before calling any of the WinAPI functions you must provide some space on the stack:

    sub rsp,40
    

    When calling other functions in x64 code on Windows you're required to provide shadow/spill/home space for that function, which is an area in which it can spill the 4 registers that are used for passing the first 4 parameters (ECX, EDX, R8, R9). Four quadwords amounts to 32 bytes (it's always 32 bytes, even if the function you call takes less than 4 arguments).

    So why subtract 40 instead of 32? There's also a requirement on stack alignment. Prior to calling a function you have to align the stack pointer to a multiple of 16 bytes (upon entering the function you'll also have RIPon the stack, making RSP % 16 == 8). So the subtraction of 8 extra bytes is because the stack pointer was 8 modulo 16 when the program started. Is that always the case? I don't know, but it seems plausible if whoever called your program's entrypoint (the OS or some runtime library) also aligned the stack before the call.

    Instead of sub rsp,40 you could use:

    and esp,0xFFFFFFF0   ; align
    sub rsp,32           ; allocate shadow space