LoadLibrary() fails when called from plugin but works in test program

Question

I'm working on a plugin for Autodesk 3ds Max 2017. As per Max's documentation, the plugin is compiled inside VS 2017 using the VS 2015 toolset against the 10.0.10586.0 Windows SDK.

When the plugin is loaded by 3ds Max, it in turn programmatically loads a number of DLLs using LoadLibrary(). Until recently, that was working fine on my machine. It's also still working fine on other developers' machines.

What happens now on my machine is that LoadLibrary() fails and returns a null handle. The error code is 127, i.e. "The specified procedure could not be found."

I am quite sure that DLLs that the one I'm trying to load to depends on are available on the system. I have triple-checked with both Dependency Walker and its modern incarnation, Dependencies.

In fact, the following program run inside Visual Studio 2017 (also using VS 2015 toolset against the 10.0.10586.0 Windows SDK) is able to load that DLL just fine:

#include <Windows.h>

int main(int argc, char* argv[])
{
    auto result = LoadLibrary(L"C:\\path\\to\\appleseed.dll");
}

To investigate this problem further, I've turned to GFlags which is part of the Debugging Tools for Windows 10.

Here is the full debugger output emitted when calling LoadLibrary(): https://gist.github.com/dictoon/b3f9f7cb52d2d81078965133d5035a03

I believe (but I'm not 100% sure) that the relevant error from this output is the following:

5f4c:54bc @ 131346281 - LdrpReportError - ERROR: Locating export "StackWalkEx" for DLL "C:\Windows\SYSTEM32\dbgeng.dll" failed with status: 0xc0000139.
Exception thrown at 0x00007FFCC6A9EAA8 (ntdll.dll) in 3dsmax.exe: 0xC0000139: Entry Point Not Found.
5f4c:54bc @ 131346281 - LdrpGenericExceptionFilter - ERROR: Function LdrpSnapModule raised exception 0xc0000139
    Exception record: .exr 000000000223F030
    Context record: .cxr 000000000223EB40

dbgeng.dll appears to be Visual Studio's debugger so I thought that maybe the error would only occur when 3ds Max was run inside Visual Studio with the debugger attached but that's not the case: the DLL also fails to load when 3ds Max is launched on its own.

Inspecting C:\Windows\SYSTEM32\dbgeng.dll with Dependencies reveals that it does export a StackWalkEx() symbol:

Edit 1: @RbMm commented that StackWalkEx() is actually not exported by dbgeng.dll but by dbghelp.dll as the screenshot below shows:

Edit 2: With a release build of the DLL I'm trying to load, everything works fine too. Looking at the debugger's output, no attempt is made at loading dbgeng.dll or finding StackWalkEx().

Edit 3: It appears that the debug version of the library I'm trying to load does have a dependency toward dbgeng.dll:

But a small test DLL written from scratch doesn't!

Edit 4: I now suspect that Boost 1.69 (on which the DLL I'm trying to load depends) to which we recently switched (from Boost 1.55) has introduced a dependency on dbgeng.dll via its Stacktrace library introduced in Boost 1.65.

Answer 1

The problem is an older version of dbghelp.dll which ships with 3ds Max and conflicts with the newer version the appleseed.dll has been linked with during the debug build.

The call to the function StackWalkEx, which failed, has a dependency on dbghelp.dll StackWalkEx function

The solution is to inactivate or replace the older, incompatible version of dbghelp.dll inside the Max root directory.

StackWalkEx is part of DbgHelp and used by System Services. It may have been updated recently during a Windows update causing the breaking change.