An old game doesn't work on my PC and I want to fix it. A function of the game might cause the problem. So I decided to hook this function with Detours. The following code is generated by IDA. mbstowcs is the function I want to hook.
.text:0043A9D4 ; int __fastcall mbstowcs(LPWSTR lpWideCharStr, LPCSTR lpMultiByteStr)
.text:0043A9D4 mbstowcs proc near ; CODE XREF: sub_408D58+39↑p
.text:0043A9D4 ; sub_408D58+65↑p ...
.text:0043A9D4
.text:0043A9D4 cchWideChar = dword ptr -14h
.text:0043A9D4
.text:0043A9D4 push ebx
.text:0043A9D5 push esi
.text:0043A9D6 push edi
.text:0043A9D7 push ebp
.text:0043A9D8 push ecx
.text:0043A9D9 mov [esp+14h+cchWideChar], ecx
.text:0043A9DC mov esi, edx
.text:0043A9DE mov edi, eax
.text:0043A9E0 test edi, edi
.text:0043A9E2 jz short loc_43A9F1
.text:0043A9E4 cmp [esp+14h+cchWideChar], 0
.text:0043A9E8 jnz short loc_43A9F1
.text:0043A9EA xor eax, eax
.text:0043A9EC jmp loc_43AA77
.text:0043A9F1 ; ---------------------------------------------------------------------------
.text:0043A9F1
.text:0043A9F1 loc_43A9F1: ; CODE XREF: mbstowcs+E↑j
.text:0043A9F1 ; mbstowcs+14↑j
.text:0043A9F1 test edi, edi
.text:0043A9F3 jz short loc_43AA5D
.text:0043A9F5 mov ebx, [esp+14h+cchWideChar]
.text:0043A9F8 push ebx ; cchWideChar
.text:0043A9F9 push edi ; lpWideCharStr
.text:0043A9FA push 0FFFFFFFFh ; cbMultiByte
.text:0043A9FC push esi ; lpMultiByteStr
.text:0043A9FD push 9 ; dwFlags
.text:0043A9FF push 0 ; CodePage
.text:0043AA01 call MultiByteToWideChar
.text:0043AA06 test eax, eax
.text:0043AA08 jz short loc_43AA0D
.text:0043AA0A dec eax
.text:0043AA0B jmp short loc_43AA77
At line .text:0043A9DE, a parameter seems to be passed via the EAX register. As far as I know, the target function and detour function must have same calling convention. The problem is, I don't know what calling convention requires that parameter be passed via EAX.
I'm using MSVC 19.40.
I tried to use the default and fastcall calling conventions, but no parameters are passed via EAX.
How to hook this function? How to define a function whose parameter is passed through EAX?
It works. I defined a function like this:
size_t __attribute__((regparm(3))) better_mbstowcs(wchar_t* pwcs, const char* s, size_t n)
and compiled it with clang-cl. IDA output:
.text:10001060 public better_mbstowcs
.text:10001060 better_mbstowcs proc near ; DATA XREF: DllMain(x,x,x)+3C↓o
.text:10001060 ; DllMain(x,x,x)+69↓o ...
.text:10001060 push esi
.text:10001061 test eax, eax
.text:10001063 jz short loc_1000106B
.text:10001065 xor esi, esi
.text:10001067 test ecx, ecx
.text:10001069 jz short loc_10001088
.text:1000106B
.text:1000106B loc_1000106B: ; CODE XREF: better_mbstowcs+3↑j
.text:1000106B test eax, eax
.text:1000106D cmovz ecx, eax
.text:10001070 push ecx ; cchWideChar
.text:10001071 push eax ; lpWideCharStr
.text:10001072 push 0FFFFFFFFh ; cbMultiByte
.text:10001074 push edx ; lpMultiByteStr
.text:10001075 push 0 ; dwFlags
.text:10001077 push 0 ; CodePage
.text:10001079 call ds:MultiByteToWideChar
It looks good.