Why does my low level windows key hook stop working?

Question

This is a continuation of my original question Why is D3D10SDKLayers.dll loaded during my DX11 game? I am creating a DX11 game and am using a low-level windows key hook to capture Alt+Enter so that I can toggle fullscreen using my own methods instead of having Windows do it automatically, which inevitably causes problems. A description of this process and details are available in the linked question. My problem is that the key hook consistently stops working after the 6th Alt+Enter for some reason. I am not unregistering it myself.

Here is the key hook code:

LRESULT _stdcall MyClass::WindowsKeyHook( s32 nCode, WPARAM wParam, LPARAM lParam ) {
    printf("Key hook called, nCode: %d. ", nCode);
    if( nCode < 0 || nCode != HC_ACTION )  { // do not process message 
        return CallNextHookEx( MyClassVar.GetWindowsKeyHook(), nCode, wParam, lParam );
    }
    printf(" Key hook status ok.\n");

    BOOL bEatKeystroke = FALSE;
    KBDLLHOOKSTRUCT* p = ( KBDLLHOOKSTRUCT* )lParam;
    switch( wParam ) {
        //NOTE: Alt seems to be a system key when it is PRESSED, but a regular key when it is released...
        case WM_SYSKEYDOWN:
            if(p->vkCode == VK_MENU || p->vkCode == VK_LMENU || p->vkCode == VK_RMENU) {
                MyClassVar.SetAltPressed(TRUE);
            }
            if(MyClassVar.IsAltPressed() && p->vkCode == VK_RETURN) {
                bEatKeystroke = TRUE;
                MyClassVar.SetAltEnterUsed(TRUE);
                printf("Alt+Enter used.\n");
            }
            break;
        case WM_SYSKEYUP:
            //NOTE: releasing alt+enter causes a SYSKEYUP message with code 0x13: PAUSE key...
            break;
        case WM_KEYDOWN:
            break;
        case WM_KEYUP: {
            if(p->vkCode == VK_MENU || p->vkCode == VK_LMENU || p->vkCode == VK_RMENU) {
                MyClassVar.SetAltPressed(FALSE);
            }
            bEatKeystroke = ( !MyClassVar.IsShortcutKeysAllowed() &&
                                ( p->vkCode == VK_LWIN || p->vkCode == VK_RWIN ) );
            break;
        }
    }

    if( bEatKeystroke ) {
        return 1;
    }
    else {
        return CallNextHookEx( MyClassVar.GetWindowsKeyHook(), nCode, wParam, lParam );
    }
}

If you need more information, just tell me what is needed. I have no idea why this is happening, so I'm not sure what kind of information I need to provide. The only way to get rid of a key hook besides unregistering it explicitly is if Windows times it out, as far as I know. All MyClassVar methods are inline to be as fast as possible, and Alt+Enter is handled from a separate thread.

Answer 1

The only way to get rid of a key hook besides unregistering it explicitly is if Windows times it out, as far as I know.

That would be correct, and you might be able to confirm it's timing out by increasing the LowLevelHooksTimeout registry key temporarily.

It's also possible, but wildly unlikely that another keyboard hook's gotten there first and is swallowing all the input before your hook is called (which is exactly what your hook tries to do with certain key combinations).

From your comment, it sounds like you want to pick a full-screen resolution when DXGI switches to full-screen for you.

The following is an excerpt from DirectX Graphics Infrastructure (DXGI): Best Practices. This information may or may not help, as will the content I've elided prior to this excerpt which talks about WM_SIZE.

The methodology of the preceding explanation follows a very particular path. DXGI set the full-screen resolution to the desktop resolution by default. Many applications, however, switch to a preferred full-screen resolution. In such a case, DXGI provides IDXGISwapChain::ResizeTarget. This should be called before calling SetFullscreenState. Although these methods can be called in the opposite order (SetFullscreenState first, followed by ResizeTarget), doing so causes an extra WM_SIZE message to be sent to the application. (Doing so can also cause flickering, since DXGI could be forced to perform two mode changes.) After calling SetFullscreenState, it is advisable to call ResizeTarget again with the RefreshRate member zeroed out. This amounts to a no-operation instruction in DXGI, but it can avoid issues with the refresh rate, which are discussed next.

The DXGI Overview has some more information on the matter, that ResizeTarget might not be as helpful as one would hope it would be:

By default, DXGI chooses the output that contains most of the client area of the window. This is the only option available to DXGI when it goes full-screen itself in response to alt-enter.

However, it does mention that that the size is determined by the client area of the window. Perhaps you want to limit the client area, instead of installing a keyboard hook?