Search code examples
c++visual-studio-2012comdirectshow

Memory corrupted in CBaseVideoRenderer-derived class


I try to write a DirectShow renderer that is able to capture VIDEOINFO2-based streams in a VS2012 project. To do so, I took the baseclasses project from the Windows 7 SDK, upgraded it to VS2012 and built it. My renderer looks like

class CapturingVideoRenderer : public CBaseVideoRenderer {

public:
    CapturingVideoRenderer(LPUNKNOWN pUnk, HRESULT *phr);
    virtual ~CapturingVideoRenderer(void);
    virtual HRESULT CheckMediaType(const CMediaType *pmt);
    virtual HRESULT DoRenderSample(IMediaSample *pMediaSample);
    virtual HRESULT SetMediaType(const CMediaType *pmt);
    virtual HRESULT StartStreaming(void);

protected:
    typedef CBaseVideoRenderer Base;

public:
    DECLARE_IUNKNOWN;

};

From the implementation, the important part is the constructor:

CapturingVideoRenderer::CapturingVideoRenderer(LPUNKNOWN pUnk, HRESULT *phr)
        : Base(::CLSID_CapturingVideoRenderer, Name_CapturingVideoRenderer,
        pUnk, phr) {
}

If I now create an instance (I do it directly for debugging), the application crashes:

IBaseFilter *filter = nullptr;
CapturingVideoRenderer *renderer = nullptr;
HRESULT hr = S_OK;

obj = new CapturingVideoRenderer(nullptr, &hr);
if (FAILED(hr)) { [...] }

//hr = obj->NonDelegatingQueryInterface(IID_IBaseFilter,
//    reinterpret_cast<void **>(&filter));
hr = obj->QueryInterface(IID_IBaseFilter,
    reinterpret_cast<void **>(&filter));
if (FAILED(hr)) { [...] }

The problem seems to be that the vtable (or more) gets somehow corrupted when the code between the last base ctor and my ctor is executed. If I call NonDelegatingQueryInterface directly (commented out), it works (but crashes later), but the call to QueryInterface crashes as it gets the actual implementation from the CUnkown base class, which is corrupted.

I have also stepped through the disassembly although I have little knowledge about assembler languages. Before the body of my ctor is entered, the following instructions are executed, which overwrite a part of the base class (CUnknown's reference counter at located at rax+18h as well as other fields of the base class):

000000013FCBABEF  mov         rax,qword ptr [this]
000000014000AC07  lea         rcx,[CapturingVideoRenderer::`vftable' (0140147A28h)]  
000000014000AC0E  mov         qword ptr [rax+18h],rcx 
000000013FCBAC02  mov         rax,qword ptr [this]  
000000013FCBAC0A  lea         rcx,[CapturingVideoRenderer::`vftable' (013FDF7A38h)]  
000000013FCBAC11  mov         qword ptr [rax+20h],rcx  
000000013FCBAC15  mov         rax,qword ptr [this]  

I already ran VS2012's static code analysis, but it did not find anything useful (some wrong SAL annotations in the baseclasses library from the Windows SDK).

The most important question is of course: how can I fix this?

My other questions are: Are there any known problems using the DirectShow base classes in VS2012? Is it possible at all? Do I have to change some compiler settings (atm, everything is the same as in the project for the executable, which basically comes directly from the wizard - platform toolset v110 and Unicode)?


Solution

  • It seems that you just cannot debug a DirectShow filter this way. It works fine if I register it "normally", so COM seems to do some important things here...