Search code examples
c++visual-c++gcccalling-convention

Calling convention between MSVC and gcc


I have a program build with MSVC which is loading a dll dynamically. The dll provides a function which is called from the main program. If both are build with either MSVC or gcc everything is fine, but when I compile e.g. main with MSVC and the dll with gcc something is wrong.

#  ifdef __GNUC__
#    define CDECL __attribute__ ((__cdecl__))
#  else
#    define CDECL __cdecl
#  endif

struct EXP result {
    uint32_t code;
};

#define SUCCESS result{0};

virtual result CDECL foo(char const* const*& target) const {
    target = (char const* const*)0xAFFE;
    return SUCCESS;
}

The problem is, after the call target is zero instead of 0xAFFE. The main program is compiled using __cdecl as calling convention. The struct is packed (no alignment), but I also tried to align to different sizes (1, 2, 4, 8, 16). I also tried to use __declspec/__atribute__(dllexport) and different combinations of both variants.

If I take a look into the assembler code, there are two big differences:

; MSVC                          |   gcc
;===============================|================================
; before calling                |
;-------------------------------|--------------------------------
                                |   sub     dword ptr [esp+4],8
                                |
; foo();                        |
;-------------------------------|--------------------------------
push    ebp                     |   push    ebp
mov     ebp,esp                 |   mov     ebp,esp
mov     eax,dword ptr [target]  |   
mov     dword ptr [eax],0AFFEh  |   
mov     eax,dword ptr [ebp+0Ch] |   mov     eax,dword ptr [ebp+0Ch]
mov     dword ptr [eax],0       |   mov     dword ptr [eax],0AFFEh
                                |   mov     eax,0
pop     ebp                     |   pop     ebp
ret                             |   ret

Why is this even if I use the same calling conventions on both compilers ? And how do I do fix it?


Solution

  • Even if it took me one year to answer my own question (I'm a realy lazy person ;) , I would like to clarify this. As Puppy mentioned in his answer, the ABI between the compilers differs. But this is just half true. Since Microsoft invented COM back in 1992 they created an interface which can be mapped into c++ vtable interfaces. So other big compiler vendors implemented the mapping between COM and C++ vtables:

    [...] Since a Windows compiler that can't use COM is pretty limited, other compiler vendors enforced the mapping between COM vtables and C++ vtables. [...]

    Also as Remy Lebeau mentioned in his comments above:

    The function is returning a structure. More likely, msvc and gcc do not agree on how to pass that structure around, whether on the call stack or in registers , etc. Returning a struct from a function is not portable. The calling convention dictates how parameters are passed, but does not dictate how non-trivial return values are passed. The portable solution is to either return just the uint32_t by itself, or else pass the struct into the function as a pointer parameter and let the function fill it as needed.

    [...] returning just a pointer by itself would be covered by the rules of the calling convention. Trivial built-in types (integers, floats/doubles, pointers, etc) are covered. User-defined types (classes/structs) are at the discretion of each compiler, typically due to differences in how they optimize their code.

    So, as the return value is not covered by the rules of the convention, only trivial types should be used as return values.

    Mentionable readings: