Search code examples
c++windowswinapihookdetours

Hooking Windows API function crashes application

I'm making a DLL that hooks specific Windows API functions with Microsoft Detours, in order to perform some analysis. The DLL is added to a third party's game's import table, in order to have the DllMain function run, and the hooks applied.

I ran into problems when I tried to hook the GetFileSize function. When I do a detour, even a simple one, the game crashes almost immediately with an access violation exception. It seems to me that the detour corrupts the application in some way. Here's the simplest code that causes the problem in my case:

static DWORD(WINAPI * TrueGetFileSize) (

    HANDLE  hFile,
    LPDWORD lpFileSizeHigh

    ) = GetFileSize;


DWORD GetFileSizeDetour(HANDLE hFile, LPDWORD lpFileSizeHigh) {
    return TrueGetFileSize(hFile, lpFileSizeHigh);
}


BOOL WINAPI DllMain(HINSTANCE hinstDLL, DWORD dwReason, LPVOID lpvReserved)
{
    if (DetourIsHelperProcess()) {
        return TRUE;
    }

    if (dwReason == DLL_PROCESS_ATTACH) {

        DetourRestoreAfterWith();

        DetourTransactionBegin();
        DetourUpdateThread(GetCurrentThread());
        DetourAttach(&(PVOID&)TrueGetFileSize, GetFileSizeDetour);
        DetourTransactionCommit();
    }
    else if (dwReason == DLL_PROCESS_DETACH) {
        DetourTransactionBegin();
        DetourUpdateThread(GetCurrentThread());
        DetourDetach(&(PVOID&)TrueGetFileSize, GetFileSizeDetour);
        DetourTransactionCommit();
    }
    return TRUE;
}

When I debug the game, with the added DLL, the detour is actually hit once, but when I continue execution, it fails. Here's a screenshot of the call stack window in Visual Studio when the process crashes:

Crash call stack

The problem seems to occur during initialisation of the CRT by dsound.dll, which I guess is how Windows games handle sound.

The process crashes with an access violation exception, saying it can't execute the location in the call stack screenshot.

This makes it seem like the detour is somehow corrupting the application, making some code dependant on the function fail. However it is probably much more likely that I'm misunderstanding something, or trying to do something the wrong way. :)

Any help or guidance will be much appreciated!

Solution

  • Problem

    The calling convention of your detour function

    DWORD GetFileSizeDetour(HANDLE hFile, LPDWORD lpFileSizeHigh)

    does not match the calling convention of the original GetFileSize() function:

    DWORD WINAPI GetFileSize( HANDLE hFile, LPDWORD lpFileSizeHigh );

    I've taken the latter from "fileapi.h" via right click > "go to declaration" and removed the irrelevant stuff.

    WINAPI is a define for __stdcall. Your detour function has no explicit calling convention modifier, so MSVC uses __cdecl by default.

    Why is this a problem? __cdecl requires the caller to cleanup the stack, whereas __stdcall requires the called function to cleanup the stack. Obviously the stack can be damaged, if incorrect calling convention is used.

    Bonus Reading: Calling Conventions Demystified

    Solution

    Fix the calling convention by adding WINAPI:

     DWORD WINAPI GetFileSizeDetour(HANDLE hFile, LPDWORD lpFileSizeHigh)