Direct assembly code execution in VBA - only fails in Excel

I have some VBA code to swap one of the function pointers in a COM vtable with a NO-OP bit of assembly code. It works in twinBASIC which is a standalone VBA environment, so I know I'm very close, however in Excel it crashes.

This is the minrepro, works in tB and in theory not in Excel VBA.

Class DummyThing
    Sub DoNothing()
    End Sub
End Class

Module VBA
    Private Const MEM_COMMIT As Long = &H1000
    Private Const MEM_RESERVE As Long = &H2000
    Private Const PAGE_EXECUTE_READWRITE  As Long = &H40

    Declare PtrSafe Function VirtualAlloc Lib "kernel32" ( _
            ByVal lpAddress As LongPtr, _
            ByVal dwSize As Long, _
            ByVal flAllocationType As Long, _
            ByVal flProtect As Long) As LongPtr

    Sub Main()
        Dim code(1 To 4) As Byte
        code(1) = CByte(&h48)    
        code(2) = CByte(&h31)
        code(3) = CByte(&hC0) 'xor rax, rax to clear it - this is like setting the hresult to 0
        code(4) = CByte(&hC3) 'ret

        Dim buffer As LongPtr
        buffer = VirtualAlloc(0&, UBound(code) - LBound(code) + 1, MEM_COMMIT Or MEM_RESERVE, PAGE_EXECUTE_READWRITE)

        If buffer = 0 Then
            Debug.Print "VirtualAlloc() failed (Err:" ; Err.LastDllError ; ")."
            Exit Sub
        End If

        CopyMemory ByVal buffer, code(1), UBound(code) - LBound(code) + 1

        Dim base As DummyThing
        Set base = New DummyThing

        Dim vtable As LongPtr
        vtable = MemLongPtr(ObjPtr(base))

        MemLongPtr(vtable + PTR_SIZE * 7) = buffer

        base.DoNothing 'Excel VBA crashes here, tB prints the message below
        Debug.Print vbNewLine ; "Done!"
    End Sub

End Module

_{The memory apis come from here https://github.com/cristianbuse/VBA-MemoryTools/blob/master/src/LibMemory.bas}

It is a super simple 64 bit assembly code that runs by swapping the vtable of Sub DoNothing() out and replacing it with a pointer to some executable opcodes. The assembly code is nothing more than

48 31 C0          xor    rax, rax                ; Set return value to 0
C3                ret                            ; Return

What might be causing the crash - maybe VBA checks the vtable integrity and that it points to memory in an expected address range? But I've never had this issue before with overloading vtables.

Solution

Found an answer

In VBA64, for classes implemented by VBA, [the interpreter] doesn't use the vtable function pointers directly. Instead, it detects that the object is implemented by VBA (itself) and shortcuts to the direct pcode implementation of the requested member instead of using the native function pointer which has to jump back into the pcode virtual machine. It's an optimization, though in reality it's a micro-optimization, but ultimately it prevents simple vtable patching from working.

It wasn't always this way. In the early versions of VBA64 (office 2010 without service packs), they did use the native vtable function calls, and you could at that time use normal vtable patching.

https://discord.com/channels/927638153546829845/1157647794484543589/1157660431545024553

So the crash comes from VBA seeing the class is a VBA class and then using the pointer from the vtable as a signpost to shortcut into pcode as an optimisation.

Except because we overwrote the function pointer, it doesn't have corresponding pcode in the same place so the lookup fails or VBA starts trying to interpret random memory as pcode leading to the crash.

Solution is invoke the assem without vtable patching to avoid the optimisation - e.g. using dispcallfunc, or by implementing the entire COM object manually with CoTaskMemAlloc and a handcrafted vtable.