I have weird problem when tracing my Media Foundation application using mftrace: when running in mftrace, the application crashes with an access violation. It is running from the Visual Studio debugger, but I need to get mftrace running to troubleshoot a Media Foundation problem. After investigating the problem for quite some time, I have added my own tracing code and discovered the following:
2952 22320,5CE8 13:05:14.78185 CMFStreamSinkDetours::GetEvent @000001CFB1E85BD0 - enter
2953 22320,5CE8 13:05:14.78455 CKernel32ExportDetours::OutputDebugStringA @ [23784 ENTER]RtspStreamSinkBase<class RtspVideoStreamSink>::IsMediaTypeSupported
2954 __M_F_T_R_A_C_E___LOG__
The method RtspStreamSinkBase<class RtspVideoStreamSink>::IsMediaTypeSupported is the one actually crashing due to an invalid (but non-null) pointer passed in as media type. I know that from attaching the debugger to the crashed process.
For me, it looks like as if mftrace thinks it is calling GetEvent, but the method actually invoked is IsMediaTypeSupported.
Has anyone encountered such an issue before? What was the solution? What could cause such an issue?
Best regards, Christoph
Update: I did some further Research and found two possible areas where it might go wrong: first, I am using ATL for my implementation (not only ATL::CComPtr, but also the other base classes like ATL::CCoClass for using the registrar etc.). Although I have worked with detours myself, I do not know how mftrace uses it. I assume that they are overwriting the vtable pointers.
Could it be that using ATL prevents the detours from being installed correctly? Is it, in principle, possible to use ATL for implementing MFT classes? Has anyone done this successfully (most samples I see in the internet do all the COM stuff manually)?
The second observation is that only the debug build crashes in mftrace. It therefore might be that some unintended side effect in debug code (eg assert(ptr = nullptr)) is causing the issue. I have not found one in my own code so far, though.
Could it be that the reference tracking in ATL (#define _ATL_DEBUG_INTERFACES) interferes with detours/mftrace?
I finally found the problem: as it turns out, enabling #define _ATL_DEBUG_INTERFACES is a very bad idea with ATL-based MF classes. After I removed this define, the internal error disappeared and the trace does not crash any more.