How to write x64 machine code into virtual memory and execute it for Windows in C++

I have been wondering how V8 JavaScript Engine and any other JIT compilers execute the generated code.

Here are the articles I read during my attempt to write a small demo.

I only know very little about assembly, so I initially used http://gcc.godbolt.org/ to write a function and get the disassembled output, but the code is not working on Windows.

I then wrote a small C++ code, compiled with -g -Og, then get disassmbled output with gdb.

#include <stdio.h>

int square(int num) {
    return num * num;
}

int main() {
    printf("%d\n", square(10));
    return 0;
}

Output:

Dump of assembler code for function square(int):
=> 0x00000000004015b0 <+0>:     imul   %ecx,%ecx
   0x00000000004015b3 <+3>:     mov    %ecx,%eax
   0x00000000004015b5 <+5>:     retq

I copy-pasted the output ('%' removed) to online x86 assembler and get { 0x0F, 0xAF, 0xC9, 0x89, 0xC1, 0xC3 }.

Here is my final code. if I compiled it with gcc, I always get 1. If I compiled it with VC++, I get random number. What is going on?

#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include <windows.h>

typedef unsigned char byte;
typedef int (*int0_int)(int);

const byte square_code[] = {
    0x0f, 0xaf, 0xc9,
    0x89, 0xc1,
    0xc3
};

int main() {
    byte* buf = reinterpret_cast<byte*>(VirtualAlloc(0, 1 << 8, MEM_RESERVE | MEM_COMMIT, PAGE_READWRITE));
    if (buf == nullptr) return 0;
    memcpy(buf, square_code, sizeof(square_code));
    {
        DWORD old;
        VirtualProtect(buf, 1 << 8, PAGE_EXECUTE_READ, &old);
    }
    int0_int square = reinterpret_cast<int0_int>(buf);
    int ans = square(100);
    printf("%d\n", ans);
    VirtualFree(buf, 0, MEM_RELEASE);
    return 0;
}

Note

I am trying to learn how JIT works, so please do not suggest me to use LLVM or any library. I promise I will use a proper JIT library in real project rather than writing from scratch.

Solution

Note: as Ben Voigt points out in the comments, this is really only valid for x86, not x86_64. For x86_64 you just have some errors in your assembly (which are still errors in x86 as well) as Ben Voigt points out as well in his answer.

This is happening because your compiler could see both sides of the function call when you generated your assembly. Since the compiler was in control of generating code for both the caller and the callee, it didn't have to follow the cdecl calling convention, and it didn't.

The default calling convention for MSVC is cdecl. Basically, function parameters are pushed onto the stack in the reverse of the order they're listed, so a call to foo(10, 100) could result in the assembly:

push 100
push 10
call foo(int, int)

In your case, the compiler will generate something like the following at the call site:

push 100
call esi ; assuming the address of your code is in the register esi

That's not what your code is expecting though. Your code is expecting its argument to be passed in the register ecx, not the stack.

The compiler has used what looks like the fastcall calling convention. If I compile a similar program (I get slightly different assembly) I get the expected result:

#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include <windows.h>

typedef unsigned char byte;
typedef int (_fastcall *int0_int)(int);

const byte square_code[] = {
    0x8b, 0xc1,
    0x0f, 0xaf, 0xc0,
    0xc3
};

int main() {
    byte* buf = reinterpret_cast<byte*>(VirtualAlloc(0, 1 << 8, MEM_RESERVE | MEM_COMMIT, PAGE_READWRITE));
    if (buf == nullptr) return 0;
    memcpy(buf, square_code, sizeof(square_code));
    {
        DWORD old;
        VirtualProtect(buf, 1 << 8, PAGE_EXECUTE_READ, &old);
    }
    int0_int square = reinterpret_cast<int0_int>(buf);
    int ans = square(100);
    printf("%d\n", ans);
    VirtualFree(buf, 0, MEM_RELEASE);
    return 0;
}

Note that I've told the compiler to use the _fastcall calling convention. If you want to use cdecl, the assembly would need to look more like this:

push ebp
mov  ebp, esp
mov  eax, DWORD PTR _n$[ebp]
imul eax, eax
pop  ebp
ret  0

(DISCLAMER: I'm not great at assembly, and that was generated by Visual Studio)