How to hook WndProc in WINUI

Question

Over the years, I've hooked the WndProc in everything from VB3 to C# in WinForms with no issue, but WINUI-3 and C# is giving me problems.

My DLL Imports are:

        /// <summary>
        /// Sets window data.
        /// </summary>
        /// <param name="hWnd">The handle of the window to set.</param>
        /// <param name="nIndex">The index of the item to set.</param>
        /// <param name="dwNewLong">The new value.</param>
        /// <returns></returns>
        [DllImport("user32.dll",
            EntryPoint = "SetWindowLongPtr",
            CallingConvention = CallingConvention.Cdecl)]
        private static extern IntPtr SetWindowLongPtr64(HandleRef hWnd, int nIndex, IntPtr dwNewLong);

        // Defines our window WndProcDelegate.
        private delegate IntPtr WndProcDelegate(IntPtr hwnd, uint message, IntPtr wParam, IntPtr lParam);

        // This handles calling the underlying base WndProc.
        [DllImport("user32.dll", 
            CallingConvention = CallingConvention.Cdecl,
            CharSet=CharSet.Auto)]
        static extern IntPtr CallWindowProc(WndProcDelegate lpPrevWndFunc, IntPtr hWnd, uint Msg, IntPtr wParam, IntPtr lParam);

The actual subclass/unsubclass etc, look like this:

 public IntPtr GetHwnd()
        {
            IntPtr handle = WinRT.Interop.WindowNative.GetWindowHandle(this.m_MainWindow);
            return handle;
        }

// Subclass the window.
        public void Subclass(object window)
        {
            if (this.m_isSubclassed)
            {
                return;
            }

            this.m_MainWindow = window;
            this.m_NewWndProcDelegate = new WndProcDelegate(this.ReplacementWndProc);
            IntPtr hWnd = this.GetHwnd();
            HandleRef handleRef = new(this, hWnd);
            IntPtr newDelegatePtr = Marshal.GetFunctionPointerForDelegate(this.m_NewWndProcDelegate);
            IntPtr oldDelegatePtr = SetWindowLongPtr64(handleRef, GWLP_WNDPROC, newDelegatePtr);
            this.m_OldWndProcDelegate = (WndProcDelegate)Marshal.GetDelegateForFunctionPointer(
                oldDelegatePtr,
                typeof(WndProcDelegate));
            this.m_isSubclassed = true;
        }

// Unsubclass the window.
        public void Unsubclass()
        {
            if (!this.m_isSubclassed)
            {
                return;
            }
            IntPtr hWnd = this.GetHwnd();
            SetWindowLongPtr64(new HandleRef(this, hWnd), GWLP_WNDPROC, Marshal.GetFunctionPointerForDelegate(this.m_OldWndProcDelegate));
            this.m_isSubclassed = false;
        }

        public bool IsSubclassed()
        {
            return this.m_isSubclassed;
        }


        /// <summary>
        /// This is the replacement WndProc.
        /// </summary>
        /// <param name="hwnd"></param>
        /// <param name="message"></param>
        /// <param name="wParam"></param>
        /// <param name="lParam"></param>
        /// <returns></returns>
        private IntPtr ReplacementWndProc(IntPtr hwnd, uint message, IntPtr wParam, IntPtr lParam)
        {
            // Our custom Windows messages code...
            /*if (message == WM_COPYDATA)
            {
                CopyDataStruct copyStruct = (CopyDataStruct)Marshal.PtrToStructure(lParam, typeof(CopyDataStruct));
                string messageText = Utf8PtrToString(copyStruct.lpData);
                int cbData = copyStruct.cbData;
                // Do something with the messageText and cbData.
            }*/

            // Finally, call the original WndProc.
            return CallWindowProc(this.m_OldWndProcDelegate, hwnd, message, wParam, lParam);
        }

The issue I have is a nebulous SystemEngineException (-2146233082) error, with all null information.

So, something has clearly gone wrong in my implementation. Is there a tried/tested way to do this with WINUI?

For anyone wondering why I'm trying to do this, I need to handle a WM_COPYDATA message.

Many thanks in advance,

Jason

Answer 1

There's a blog article describing this exact task, and error message.

• Accessing the WndProc in a UWP Application

In that case the author was using SetWindowLongPtr to storing a pointer to the result of GetFunctionPointerForDelegate, and calling it after C# has detected the delegate is unreachable.

You are keeping the delegate alive by storing it to a member variable. GCHandle.Alloc would also have worked. So that's fine... at least in the Subclass function.

However, your Unsubclass is broken. You seem to have assumed that GetFunctionPointerForDelegate is an inverse function for GetDelegateForFunctionPointer. It's not, it creates a brand new trampoline linked to the managed delegate instance. You need to preserve the original oldDelegatePtr and put it back.

In fact, you have no reason to use GetDelegateForFunctionPointer at all, since the old wndproc is only ever passed back to WinApi functions. Just leave it as an IntPtr.