Search code examples
c++visual-studioassemblycalling-convention

MSVS 2010 C++ Compiler and Stack alignment issue?


My problem is MSVS 2010 C++ compiler is generating code in a way after returning from a function call resolved in runtime(GetProcAddress+GetModuleHandle) from another dll the compiler then tries to align stack this way:

   CALL DWORD PTR DS:[2000367C]             ;  apiresolvedinruntime.dll
   ADD ESP,12                               ;  <- this is the stack alignment

This is of course overwriting the return address and my program crashes, can someone explain me why compiler aligning the stack when it really shouldn't do it?


Solution

  • You didn't call the runtime loaded function using the correct calling convention. Calling convention specifies the default handling of what happens to the stack. Most likely, the DLL was compiled using the __stdcall calling convention (which is what e.g. the Windows DLLs use), which specifies that the called function is supposed to clean up the stack, but the calling code was declared with a function pointer using the __cdecl calling convention (which is the default). Under __cdecl, functions support variadic arguments, so the caller needs to do the cleanup of the stack, because the called function does not know how many arguments are passed.

    You need to verify that the DLL and the calling code are compiled using the same calling conventions.