Search code examples
c++visual-studio-2012visual-c++vtabledisassembly

MSVC 2012 generates different vtable pointer offsets for different files


Let's say I've got in X64 release configuration It's a an obfuscated code snippet...

// Hdr1.h
// Dozen of includes

class Cls1
{
public:
    Cls1();
    virtual void bar();
// ...
protected:
// about 7 fields where some of them are of complex template type.
bool isFlag1 : 1;
bool isFlag2 : 1;
};


// Hdr2
// Dozens of includes
class Cls2
{
public:
    // ...
    void foo();
};

I've got separate translation units to implement these classes. Say from foo I try to access virtual method of Cls1::bar and I get a crash(access violation).

void Cls2::foo()
{
   //...
   Cls1 * pCls1 = // somehow I get this goddamn pointer

   pCls1->bar(); // Here I crash
}

From disassembly I see that Cls1::Cls1 puts vtable ptr at offset 8 to the very beginning of this. From disassembly of Cls2::foo I see that it takes pointer to vtable from offset zero. Debugger is also unable to see this vtable correctly. If I manually get vtable at offset 8 - addresses appear to be correct in this table.

The question is - why could this happen, what pragma could lead to this or anything else? Compilation flags are the same for both translation units.

Below I add a bit of disassembly: This is a normal case that I face across the code:

 Module1!CSomeOkClass::CreateObjInstance:
 sub     rsp,28h
 mov     edx,4                                  ; own inlined operator new
 lea     ecx,[rdx+34h]                          ; own inlined operator new
 call    OwnMemoryRoutines!OwnMalloc (someAddr) ; own inlined operator new
 xor     edx,edx
 test    rax,rax
 je      Module1!CSomeOkClass::CreateObjInstance+0x40 (someAddr)
 **lea     rcx,[Module1!CSomeOkClass::`vftable' (someAddr)] ; Inlined CSomeOkClass::CSomeOkClass < vtable ptr**
 mov     qword ptr [rax+8],rdx                  ; Inlined CSomeOkClass::CSomeOkClass
 mov     qword ptr [rax+10h],rdx                    ; Inlined CSomeOkClass::CSomeOkClass
 mov     qword ptr [rax+18h],rdx                    ; Inlined CSomeOkClass::CSomeOkClass
 mov     byte ptr [rax+20h],dl                  ; Inlined CSomeOkClass::CSomeOkClass
 mov     qword ptr [rax+28h],rdx                    ; Inlined CSomeOkClass::CSomeOkClass
 **mov     qword ptr [rax],rcx                  ; Inlined CSomeOkClass::CSomeOkClass < offset zero**

Now let's see what I've got for Cls1::Cls1:

 Module1!Cls1::Cls1:
 mov     qword ptr [rsp+8],rbx
 push    rdi
 sub     rsp,20h
 **lea     rax,[Module1!Cls1::`vftable' (someAddress)]  ; vtable address**
 mov     rbx,rdx
 mov     rdi,rcx
 **mov     qword ptr [rcx+8],rax                ; Places at offset 8**

I assure you that Cls2 expects pointer to vtable to be at offset zero.

Compilation options are: /nologo /WX /W3 /MD /c /Zc:wchar_t /Zc:forScope /Zm192 /bigobj /d2Zi+ /Zi /Oi /GS- /GF /Oy- /fp:fast /Gm- /Ox /Gy /Ob2 /GR- /Os

I noticed that Cls1::Cls1 heavily uses SSE instructions inlined from intrinsics.

Compiler version: Microsoft (R) C/C++ Optimizing Compiler Version 17.00.50727.1 for x64

Please pay attention that this code works ok on different platforms/compilers.

I managed to figure out that the problem was in fact with this bitfield I have in the very end of Cl1 definition. The ctor generated places pointer to vtable at offset zero if I make isFlag1 + isFlag2 ordinary bools. These flags are initialized in the ctor's initializer list. By commenting out class's code one by line I narrowed down the problem to this bitfield. In order to investigate this I used WinDbg, /P compiler option, compiled cpp unit manually with the original flags provided + /FAs /Fa. It appears that it is a compiler's bug.


Solution

  • I managed to figure out that the problem was in fact with this bitfield I have in the very end of Cl1 definition. The ctor generated places pointer to vtable at offset zero if I make isFlag1 + isFlag2 ordinary bools. These flags are initialized in the ctor's initializer list. By commenting out class's code one by line I narrowed down the problem to this bitfield. In order to investigate this I used WinDbg, /P compiler option, compiled cpp unit manually with the original flags provided + /FAs /Fa. It appears that it is a compiler's bug.