Access violation when using Microsoft Detours

Question

I am having trouble with an access violation when using Microsoft Detours. I have made a dll which is loaded in to a thirdparty application. I am making a trampiline function using Detours to an undocumented function which Ida Pro displays as:

void __thiscall sub_6142E0(int a2, int a3)

My code looks like this: #include "stdafx.h" #include #include

typedef void(__stdcall* pFunc)(int d1, int d2);
pFunc FuncToDetour = (pFunc)(0x6142EC);

void MyFunc(int d1, int d2)//Function does not mach call convension __thiscall. Possible problem?
{
    printf("a2 %i, a1 %i);\n", d1, d2);
    FuncToDetour(d1, d2);
}

void Init()
{
    DetourTransactionBegin();
    DetourUpdateThread(GetCurrentThread());
    DetourAttach(&(PVOID&)FuncToDetour, MyFunc);
    DetourTransactionCommit();
}

The original assembly of the function I want to intercept is like this:

sub_6142E0 proc near

arg_0= dword ptr  8
arg_4= dword ptr  0Ch

push    ebp
mov     ebp, esp
mov     eax, [ecx+8]
mov     ecx, [ebp+arg_4]
mov     edx, [ebp+arg_0]

The alterations Detours does results in this:

.text:006142EC jmp     near ptr unk_F9C6802
...
d3d9.dll:0F9C6802 jmp     near ptr unk_F9D5FE0 //jump to function in my dll
...
void MyFunc(int d1, int d2)//my function
{
    printf("updateHealth(%i, %i);\n", d1, d2);
}
...
Stack[00004A8C]:0019FB4C sub     ah, bh
Stack[00004A8C]:0019FB4E sbb     [eax], eax //eax=0x491B -> access violation
Stack[00004A8C]:0019FB50 cmc
Stack[00004A8C]:0019FB51 inc     si
Stack[00004A8C]:0019FB53 add     [eax], dl
Stack[00004A8C]:0019FB55 add     [eax], eax
Stack[00004A8C]:0019FB57 add     [eax+80019FDh], cl
Stack[00004A8C]:0019FB5D add     byte_19FC6415[eax], dh
Stack[00004A8C]:0019FB5D ; -------------------------------------------------

The error message I get is:

The instruction 0x19FB4E referenced memory at 0x491B. The memory could not be written -> 0000491B (exc.code c0000005, tid 19084)

Answer 1

I am going to try to answer my own questing.

This boils down to mismatch in calling convention between the two functions. The function I want to hook is using __thiscall and my function is using __cdecl(default calling convention). __thiscall is used as calling convention for member functions in a class where the "this pointer" is passed in the ecx register when calling a member function.

ecx in my case is written to when MyFunc is called in order to set up the stack frame(I think). The function I hook will then get a invalid this pointer when I call it from my trampoline function.

Check this link for some explanation and examples for how this can be done correctly.