Search code examples
comvisual-studio-2017directshow

Registering a DirectshowFilter links InprocServer32 registry entry to regsvr32.exe instead of host DLL, where did I go wrong?

I have a DirectShow filter (MonogramAAC encoder) successfully compiling. It also registers successfully, however, when I try to instantiate the filter in GrapheditPlus, it shows a CO_E_ERRORINDLL. After inspecting the filter properties under GraphEditPlus, the filter was actually linked to RegSvr32.exe instead of the actual host DLL!

Here's the evidence:

enter image description here

enter image description here

I'm sure the .def file in the linker is correct, I also tested that it IS being considered by the compiler and it is correctly pointing to the filter DLL.

A little bakground info:

I couldn't get the original project to compile under Visual Studio 2017 after automated conversion. (I couldn't resolve strange MFC and runtime library linker errors between the Monogram filter and the libaac.lib library, that I'm also able to compile successfully).

Finally I decided to recreate a brand-new VS2017 Project mimicking the original one and got it to compile and register (with the only warning being the mismatch between the Project name and the output library), but obviously I missed something important while recreating the project.

Any pointers?

Solution

  • Ok, I managed to find the reason for this unexpected behavior in this old post.

    RegSvr32 just calls the DllRegisterServer entry point in your code. What gets written to the registry is entirely up to your filter. If you are using the standard baseclass call to AMovieDllRegisterServer2 as your DllRegisterServer, I would suspect that something went wrong in setting up g_hInst, which should have been done by the call DllEntryPoint in your DllMain.

    …

    The DllMain is not being called, so g_hInst stays 0 and GetModuleFileNameA(...) returns the name of the currently executing program instead of my filter.

    The filter is using MFC (dynamically linked), and its DllMain() version is not calling the DllEntryPoint() function. I had to override the MFC DllMain() by adding extern "C" { int _afxForceUSRDLL; } as described in an answer to this question and call DllEntryPoint() myself to fix the registration problem. (I also found a missing #define was the reason for the property page not showing up).