Large Vectored Exception Handler in DLL is causing program to crash without debugger

Question

I'm currently working on DLL (Windows 10, x64) which implements Vectored Exception Handler (VEH). It exports the function in sake of to be attached to an executable, nothing else. During attaching VEH it also calculates base offset with the following code snippet:

uint64_t GetModuleBase(const char *modname)
{
    HANDLE hModSnap;
    MODULEENTRY32 pe32;
    void *bs = NULL;

    // Take a snapshot of all processes in the system.
    hModSnap = CreateToolhelp32Snapshot( TH32CS_SNAPMODULE, GetCurrentProcessId() );
    if( hModSnap == INVALID_HANDLE_VALUE )
    {
        return NULL;
    }

    // Set the size of the structure before using it.
    pe32.dwSize = sizeof( MODULEENTRY32 );

    if( !Module32First( hModSnap, &pe32 ) )
    {
        CloseHandle( hModSnap );     // Must clean up the snapshot object!
        return NULL;
    }

    do
    {
        if (!StrCmpI(pe32.szModule, modname)) {
            bs = pe32.modBaseAddr;
            break;
        }
    } while( Module32Next( hModSnap, &pe32 ) );

    CloseHandle( hModSnap );

    return (uint64_t)bs;
}

Within the exception handler I'm processing only exceptions with code "0xc000001d", in other cases VEH is passing execution back to the program.

Inside handler function, which is registered in VEH I have the following structure (simplified):

#define FUNCTION_1(arg1, arg2, arg3, arg4)
    if (exception_offset == arg1) { \
        5_lines_of_code .... \
       return EXCEPTION_CONTINUE_EXECUTION; \
    }


#define FUNCTION_2(arg1, arg2, arg3, arg4)
    if (exception_offset == arg1) { \
        5_lines_of_code .... \
        return EXCEPTION_CONTINUE_EXECUTION; \
    } 


#define FUNCTION_3(arg1, arg2, arg3, arg4) 
    if (exception_offset == arg1) { \
        5_lines_of_code .... \
        return EXCEPTION_CONTINUE_EXECUTION; \
    } \

...

# 15 functions like that.

FUNCTION_1(0x192385, CPU_REGISTER1, CPU_REGISTER2);
FUNCTION_2(0x145685, CPU_REGISTER3, CPU_REGISTER4);
FUNCTION_3(0x456685, CPU_REGISTER1, CPU_REGISTER5);

# 1719 lines like that

I have warning from MSVC that: Function uses '81340' bytes of stack: exceeds /analyze:stacksize '16384'. Consider moving some data to heap.

With debugger everything works as expected. With Windbg everything works as expected.

Without debugger - application silently crashes.

Deeper investigation shows that it registers DLL, starts execution and even call handler few times, but with different exception codes (starting from 0x8..... which are debug ones), so VEH code returns on early stages. I can see same codes for the same offsets during normal application operations, btw.

And everything works perfectly while amount of that function calls is around ~1400 (if I comment part of them, doesn't matter which, just if I reduce the amount)

I've tried to:

• Comment out same function calls (to ensure it's not a function implementation bug) -> only amount matters
• Disable optimizations
• Increase stack size up to 140 Mb
• Checked for uninitialized or zeroed variables with static MSVC analysis - none of those.
• Reading internet for similar cases.
• Used file logging for debug

Nothing of that brings any effect. My guess, it's an overloaded stack.

Thus the questions:

• Is there any way to get more info from the crash without debug mode?
• Is there a way to represent such a massive amount of calls, so the code will be fast, but will not overload stack? Note: all function calls have different arguments supplied depending on offset where exception happened, thus grouping will reduce amount of calls on a very small %.
• What would be your next step should you have faced this kind of problem in order to debug it?

Thank you and hoping for your help.

Answer 1

Ok, I was able to resolve the issue:

1. Picked up different debugger - x64dbg. Amazing debugger, which attaches to launched process instead of launching it in debug mode.
2. This allowed to identify stack overflow problem during process initialization.
3. I've applied the fast fix for this, cause I'm prototyping: modified .exe file of the application with editbin /F stacksize in order to get it increased.
4. Everything works.