Search code examples
c++.netexceptionstack-overflowvisual-c++-6

.NET exception handler causing stack overflow on Visual C++ 6.0 exceptions

I have a plugin for an old application written in C++ 6.0. The files are connected in the following fashion:

  1. Starts with: C++ 6.0 .exe (third party application)
  2. loads: C++ 6.0 simple loader .dll (officially a plugin for application)
  3. loads: C++ 10.0 simple loader .dll (managed C++/CLI)
  4. loads one of: C# .NET 4.0 assembly which contains plugin
  5. loads: C++ 6.0 .dll which provides API for C# plugin to talk to application

The problem is once .NET 4.0 is loaded into the C++ 6.0 application, the next time it throws a native exception, .NET uses a vectored exception handle to handle the exception and fails spectacularly. The part that makes it really bad is that the vectored exception handler throws an exception itself, which it then tries to handle, and that fails, and it gets stuck in an infinite loop until it gets a stack overflow exception.

Here's what the stack trace looks like:

// The next 7 lines repeat until the stack overflows
clr.dll!CreateHistoryReader()
clr.dll!CreateHistoryReader()
clr.dll!GetMetaDataInternalInterfaceFromPublic()
ntdll.dll!_RtlpCallVectoredHandlers@12()
ntdll.dll!_RtlCallVectoredExceptionHanders@8()
ntdll.dll!_RtlDispatchException@8()
ntdll.dll!_KiUserExceptionDispatcher@8()
// Below is an example exception that causes this:
KernelBase.dll!RaiseException()
rpcrt4.dll!RpcRaiseException()
rpcrt4.dll!I_RpcTransConnectionFreePacket()
rpcrt4.dll!I_RpcBindingInqCurrentModifiedId()
rpcrt4.dll!NdrConformantStringMemorySize()
rpcrt4.dll!NdrComplexStructMarshall()
rpcrt4.dll!SimpleTypeMemorySize()
rpcrt4.dll!NdrClientCall2()
ole32.dll!ServerRegisterClsid(void* hRpc, void* phProcess, _RegInput* pregin, _RegOutput** ppregout unligned long* prpcstat
ole32.dll!CRpcResolver::NotifyStarted(_RegInput* pRegIn, _RegOutput** ppRegOut)
ole32.dll!CClassCache::ResumeProcessClassObjects()

There are only really 2 ways to deal with this and neither are very great:

I found with a trivial program, if I completely isolate .NET on its own thread, the non-.NET threads never run into this issue. This doesn't work in practice because the plugin API needs to make synchronous callbacks to the .NET plugin.

The other I've come up with is to iterate over every single address in memory until a call to "RemoveVectoredExceptionHandler(HANDLE)" succeeds and removes .NET's vectored exception handler. (I can speed up the search by registering my own VEH temporarily and use its handle as a starting spot). This tends to break debugging of native code.

Is there any better way to deal with this?

Solution

  • The CLR seems to have changed behaviors since I reported this issue. Since the CLR is now open source, it's possible to see what's going on under the hood.

    The CLR installs its own vectored exception handler. During vectored exception handling, it does a stack check to make sure there's enough space, unless it's a stack overflow exception. The stack space check goes wrong, and it thinks it's out of space when it's not, so it throws a stack overflow exception to unroll the stack enough to do actual work.

    I was able to trick .NET into not crashing the app by installing 2 vectored exception handlers, one before and one after. If it's the exception type that's causing the crash, I change the exception code to STACKOVERFLOW in the first handler, and change it back in the second handler. This way the CLR thinks it's a stack overflow exception and doesn't try to do stack probing.