The calling convention used in assembly differs depending to the compiler. The Question is: How ollydbg2.01 would help me to recognize the parameters passed from caller to the callee and values returned back to the caller from the callee for a CALL instruction. The assembly which I am working on is compiled by Microsoft visual C++.
Calling convention is not dependent on any compiler.
By default microsoft visual studio uses __cdecl (user has to clean up the stack)
You can force it to use __stdcall (called function is responsible for cleaning the stack)
You can also compile with __declspec (naked) ( arbitrary cleanup )
Afaik x64 is FASTCALL only (no other calling conventions are applicable)
you can recognize calling convention by looking at the assembly
if you observe
call xxxx
add esp,XXXXX
it most likely is __cdecl
if you observe
call xxxx
inside the call
leave
retn XXX then it is likely to be __stdcall
and so on
you may need to recognize a specific construct by observation of disassembly and corelate it with the calling convention
calling conventions can also be mixed arbitrarily so you may need to ascertain each function calls individually
compile with cl /Zi /nologo /W4 /analyze xxx.cpp /link /RELEASE
#include <stdio.h>
char *domain = "i am calling me" , *formatstr = "%s %s\n";
void __cdecl cdecprint (char *king) {
printf(formatstr,domain,king);
}
void __stdcall stdprint (char *king) {
printf(formatstr,domain,king);
}
void __fastcall fastprint (char *king) {
printf(formatstr,domain,king);
}
__declspec(naked) void arbitprint (char *king) {
(void)king; //unreferenced warning
__asm {
lea eax, [esp+4]
push [eax]
push domain
push formatstr
call printf
add esp,0Ch
retn
}
}
int main (void) {
cdecprint("cdecinking");
stdprint("stdking");
fastprint("fastking");
arbitprint("arbitking");
return 0x1337;
}
and the relevent disassembly of main
004010A0 callme.main PUSH EBP ; {
004010A1 MOV EBP, ESP
004010A3 PUSH callme.004121A4 ; cdecprint("cdecinking");
004010A8 CALL callme.cdecprint
004010AD ADD ESP, 4 <---------- ; call above is likely __cdecl
004010B0 PUSH callme.004121B0 ; stdprint("stdking");
004010B5 CALL callme.stdprint <--; no cleanup likely __stdcall
004010BA MOV ECX, callme.004121B8; fastprint("fastking");
004010BF CALL callme.fastprint <--; __stdcall ? No __fastcall
004010C4 PUSH callme.004121C4 ; arbitprint("arbitking");
004010C9 CALL callme.arbitprint ;
004010CE ADD ESP, 4 <--------- ; __cdecl? No __declspec (naked)
004010D1 MOV EAX, 1337 ; return 0x1337;
004010D6 POP EBP ; }
004010D7 RETN
__cdecl
00401000 callme.cdecprint PUSH EBP
00401001 MOV EBP, ESP
00401003 MOV EAX, DWORD PTR SS:[EBP+8]
00401006 PUSH EAX
00401007 MOV ECX, DWORD PTR DS:[domain]
0040100D PUSH ECX
0040100E MOV EDX, DWORD PTR DS:[formatstr]
00401014 PUSH EDX
00401015 CALL callme.printf
0040101A ADD ESP, 0C
0040101D POP EBP
0040101E RETN
__stdcall
00401020 callme.stdprint PUSH EBP
00401021 MOV EBP, ESP
00401023 MOV EAX, DWORD PTR SS:[EBP+8]
00401026 PUSH EAX
00401027 MOV ECX, DWORD PTR DS:[domain]
0040102D PUSH ECX
0040102E MOV EDX, DWORD PTR DS:[formatstr]
00401034 PUSH EDX
00401035 CALL callme.printf
0040103A ADD ESP, 0C
0040103D POP EBP
0040103E RETN 4
__fastcall
00401050 callme.fastprint PUSH EBP
00401051 MOV EBP, ESP
00401053 PUSH ECX
00401054 MOV DWORD PTR SS:[EBP-4], ECX
00401057 MOV EAX, DWORD PTR SS:[EBP-4]
0040105A PUSH EAX
0040105B MOV ECX, DWORD PTR DS:[domain]
00401061 PUSH ECX
00401062 MOV EDX, DWORD PTR DS:[formatstr]
00401068 PUSH EDX
00401069 CALL callme.printf
0040106E ADD ESP, 0C
00401071 MOV ESP, EBP
00401073 POP EBP
00401074 RETN
__declspec (naked)
00401080 callme.arbitprint LEA EAX, DWORD PTR SS:[ESP+4]
00401084 PUSH DWORD PTR DS:[EAX]
00401086 PUSH DWORD PTR DS:[domain]
0040108C PUSH DWORD PTR DS:[formatstr]
00401092 CALL callme.printf
00401097 ADD ESP, 0C
0040109A RETN
apart from the 4 above there are a few specialized calling conventions and obsolete calling conventions
__clrcall is for managed
__vectorcall is for using streaming instructions (sse 2 .....)
__thiscall used mostly oop and com methods
__pascal etc obsolete