Search code examples
winapiaslrcomctl32

Randomized Base Address - Yes (/DYNAMICBASE) and old school subclassing causes crash

I noticed an older app had the Randomized Base Address option set to No (/DYNAMICBASE:NO) for some reason. I changed it to Yes, then forgot about it. Came back days later to run the application in debug mode and found, after a few successful calls, it was crashing on a call to a the prior WNDPROC.

It had been using the old method of getting and setting GWLP_WNDPROC then calling back to the prior one CallWindowProc(OldWndProc, hwnd, message, wparam, lparam) that looked to be within comctl32.dll.

It took me a while to remember I had changed that option so I turn the option back to /DYNAMICBASE:NO and it was all working again.

In the end, I just change everything over to the SetWindowSubclass() method and it works fine with Yes (/DYNAMICBASE).

But I wonder, what was the causing the problem? Does the comctl32.dll move randomly and those old WndProc addresses are no longer valid?

TIA!!

Solution

  • The problem with the 20+ year old code was the callback was defined:

    BOOL CALLBACK MyTreeViewWndProc(HWND htv, UINT message, WPARAM wparam, LPARAM lparam)

    This was running x64 and the return values were cut off. It should be:

    LRESULT CALLBACK MyTreeViewWndProc(HWND htv, UINT message, WPARAM wparam, LPARAM lparam)

    or

    INT_PTR CALLBACK MyTreeViewWndProc(HWND htv, UINT message, WPARAM wparam, LPARAM lparam)

    The new SetWindowsSubclass() function allowed the compiler to enforce the definition for the callback parameter which was correct in the new version.