Hello World direct Assembly Code execution in VBA - so close but Access Violation on Return

Question

Context: I know it is possible to execute assembly code in vba. A simple method is to overwrite the entry of a COM object's virtual table (vtable) with a function pointer to some place in memory that contains executable instructions. Then when you invoke the COM object's overwritten method, VBA uses the standard calling convention to execute whatever function the corresponding vtable points to.

Even though I understand the theory, I have never seen this done in real life. So I'm attempting to implement a "hello world" example that just shows a message box.

Class DummyThing
    Sub DoNothing()
    End Sub
End Class

'Uses mem manip functions from https://github.com/cristianbuse/VBA-MemoryTools/blob/master/src/LibMemory.bas

Module VBA
    Private Const MEM_COMMIT = &H1000
    Private Const MEM_RESERVE = &H2000
    Private Const PAGE_READWRITE = &H4
    Private Const PAGE_EXECUTE_READWRITE = &H40

    Declare PtrSafe Function VirtualAlloc Lib "kernel32" ( _
        ByVal lpAddress As LongPtr, _
        ByVal dwSize As Long, _
        ByVal flAllocationType As Long, _
        ByVal flProtect As Long) As LongPtr

    Declare PtrSafe Function LoadLibrary Lib "kernel32" Alias "LoadLibraryA" ( _
        ByVal lpLibFileName As String) As LongPtr

    Declare PtrSafe Function GetProcAddress Lib "kernel32" ( _
        ByVal hModule As LongPtr, _
        ByVal lpProcName As String) As LongPtr
    
    [ PackingAlignment (1) ] 'byte alignment instead of dword
    Type ShellcodeStruct
        push1 As Byte
        mb_ok As Byte

        push2 As Byte
        captionAddress As Long ' Address of 'Hello World' caption

        push3 As Byte
        textAddress As Long    ' Address of 'Hello World' text

        push4 As Byte
        hwnd As Byte

        callOp As Byte
        callType As Byte
        MessageBoxWAddress As Long ' Address of MessageBoxW
        popOp As Byte              ' POP EAX

        retOp As Byte              ' RET
    End Type
    
    Sub Main()
        Dim base As DummyThing = New DummyThing
        Dim vtable As LongPtr = MemLongPtr(ObjPtr(base)) 'deref objptr to get vtable ptr
        
        Dim title As String = "foo"
        Dim caption As String = "hello"
        
        Dim MessageBoxW As LongPtr = pMessageBoxW
        Dim code As ShellcodeStruct = GetShellCode(caption, title, VarPtr(MessageBoxW))
        Dim buffer As LongPtr = VirtualAlloc(0&, LenB(code), MEM_COMMIT Or MEM_RESERVE, PAGE_EXECUTE_READWRITE)
        
        If buffer = 0 Then
            Debug.Print ("    |__ VirtualAlloc() failed (Err:" + Str(Err.LastDllError) + ").")
            Exit Sub
        Else
            Debug.Print ("    |__ VirtualAlloc() OK - Got Addr: 0x" + Hex(buffer))
        End If
        
        CopyMemory ByVal buffer, code, LenB(code)
        
        MemLongPtr(vtable + PTR_SIZE * 7) = buffer 'overwrite vtable
        base.DoNothing 'invoke the overwritten vtable
    End Sub
    
    Function pMessageBoxW() As LongPtr
        Dim hLib As LongPtr
        Dim addrMessageBoxW As LongPtr

        hLib = LoadLibrary("User32.dll")
        Return GetProcAddress(hLib, "MessageBoxW")
    End Function
    
    Function GetShellCode(ByRef caption As String, ByRef text As String, ByVal addrMessageBoxW As Long) As ShellcodeStruct
        Dim sc As ShellcodeStruct

        ' Fill in the opcodes:
        sc.push1 = &H6A
        sc.mb_ok = &H0

        sc.push2 = &H68
        ' Assuming caption is a VBA string holding 'Hello World'
        sc.captionAddress = StrPtr(caption)

        sc.push3 = &H68
        ' Assuming text is another VBA string holding 'Hello World'
        sc.textAddress = StrPtr(text)

        sc.push4 = &H6A
        sc.hwnd = &H0

        sc.callOp = &HFF
        sc.callType = &H15
        sc.MessageBoxWAddress = addrMessageBoxW
        sc.popOp = &H58
        sc.retOp = &HC3
        Return sc
    End Function

End Module

_{Note I'm using twinBASIC rather than VBA because it does not crash the host when I get an ACCESS VIOLATION. It also lets me remove packing from the UDT which is convenient. But I expect this to work in 32 bit VBA too.}

The x86 assembly code I'm trying to execute is the following:

push 0               ; MB_OK
push 'Hello World'   ; Caption (address to string in memory) - StrPtr(caption)
push 'Hello World'   ; Text (address to string in memory) - StrPtr(text)
push 0               ; hWnd
call MessageBoxW     ; VBA BSTRs are wide I think - the address is hardcoded
pop eax              ; discard return code of message box - same error without this line
ret

Problem

The code works 90% - it shows the message box with the full captions. However it then crashes with (runtime error -2147467259: NATIVE EXCEPTION: ACCESS_VIOLATION) presumably on the ret instruction.

I'm thinking about maybe the handling of the this pointer to a COM method - although that should be passed in a register and not something to worry about. Maybe there is a hresult? Any ideas how to get this working?

I'm guessing as my shellcode is being invoked as a COM method with stdcall, as the callee I need to leave the stack in a particular state and I'm not doing that correctly

Answer 1

2 mistakes:

1. The MessageBox function returns its value in eax not the stack. So pop eax pops the stack into eax; removing valid data from the stack. Incidentally we want to set eax to 0 for HRESULT_SUCCESS anyway.
2. In stdcall, the callee must clean the stack up. When VBA invokes the shellcode in the line base.DoNothing, it pushes a hidden this pointer to the COM object onto the stack. It is the job of the shellcode to consume this item from the stack.

Therefore the correct assembly instructions are:

push 0              ; MB_OK
push DWORD PTR [captionAddress]   ; Address of 'Hello World' caption
push DWORD PTR [textAddress]      ; Address of 'Hello World' text
push 0              ; hWnd
call DWORD PTR [MessageBoxWAddress] ; Direct call to MessageBoxW
xor eax, eax        ; Clear EAX register, essentially setting HRESULT to S_OK (0)
ret 4               ; Return, and adjust stack by 4 bytes (for the "this" pointer)

Note ret 4 will remove the 4 byte this pointer from the stack, since we didn't consume it anywhere else. xor eax, eax sets the HRESULT (I think) to 0