Subscribing to windows messages through unmanaged c++ dll from c# net core

Question

I am trying to subscribe to windows message events/messaging system from c# net core through unamanged c++ dll using pinvoke.

Issues I am having.

Getting the handle for my process or creating an empty window (does .net even support that).

  var hwnd = Process.GetCurrentProcess().Handle;
  var hwnd1 = Process.GetCurrentProcess().Handle.ToPointer();

Is either of that is valid to get the handle.

How do I marshal that handle to c++ HWND type. IntPtr seems like obvious choice, but it does not work.

Here is what I am using to subscribe to events

public class MsgSubscribe : IDisposable
{
    private readonly Importer _importer;

    [UnmanagedFunctionPointer(CallingConvention.Winapi)]
    private delegate Status DMsgSubscribe(uint msgType, uint msgQ,  int hwnd, uint msgId);
    private static DMsgSubscribe _dMsgSubscribe;
    private IntPtr PMsgSubscribe { get; set; }

    public bool Available { get; set; }

    public MsgSubscribe(Importer importer)
    {
        _importer = importer;

        if (_importer.hCurModule != IntPtr.Zero)
        {
            PMsgSubscribe = Importer.GetProcAddress(_importer.hCurModule, "MsgSubscribe");
            Available = PUlyMsgSubscribe != IntPtr.Zero;
        }
    }

    public Status MsgSubscribe(uint msgType, uint msgQ, int hwnd, uint msgId)
    {
        Status result = Status.FunctionNotAvailable;


        if (Available)
        {
            _dMsgSubscribe = (DMsgSubscribe)Marshal.GetDelegateForFunctionPointer(PMsgSubscribe, typeof(DMsgSubscribe));
            result = _dMsgSubscribe(msgType, msgQ, hwnd, msgId);
        }

        return result;
    }



    public void Dispose()
    {
    }
}

I've tried IntPtr and int for HWND marshalling, neither works. Also I am not sure how I am supposed to catch window message based events, there is very little online if anything.

Any help appreciated.

Answer 1

Eventually found a way to make this work, it involves creating a window through c++ pinvoke.

using System;
using System.Collections.Generic;
using System.Runtime.InteropServices;
using System.Threading;
using System.Threading.Tasks;
using Constants.Constants;
using Constants.Enums;
using Models.WindowsApiModels;

namespace Dependencies.MessagingHandling
{
    public class CustomWindow : IDisposable
    {
        delegate IntPtr WndProc(IntPtr hWnd, uint msg, IntPtr wParam, IntPtr lParam);


        private const int ErrorClassAlreadyExists = 1410;
        public IntPtr Handle { get; private set; }
        public List<YourType> Messages { get; set; }

        public void Dispose()
        {
            if (Handle != IntPtr.Zero)
            {
                Importer.DestroyWindow(Handle);
                Handle = IntPtr.Zero;
            }
        }

        public CustomWindow()
        {
            Messages = new List<YourType>();
            var className = "Prototype Messaging Class";

            WndProc mWndProcDelegate = CustomWndProc;

            // Create WNDCLASS
            WNDCLASS windClass = new WNDCLASS
            {
                lpszClassName = className,
                lpfnWndProc = Marshal.GetFunctionPointerForDelegate(mWndProcDelegate)
            };

            UInt16 classAtom = Importer.RegisterClassW(ref windClass);

            int lastError = Marshal.GetLastWin32Error();

            if (classAtom == 0 && lastError != ErrorClassAlreadyExists)
            {
                throw new Exception("Could not register window class");
            }

            // Create window
            Handle = Importer.CreateWindowExW(
                0,
                className,
                "Prototype Messaging Window",
                0, 0, 0, 0, 0,
                IntPtr.Zero,
                IntPtr.Zero,
                IntPtr.Zero,
                IntPtr.Zero
            );
        }

        private IntPtr  CustomWndProc(IntPtr hWnd, uint msg, IntPtr wParam, IntPtr lParam)
        {
            //handle your message here

            return Importer.DefWindowProc(hWnd, msg, wParam, lParam);
        }



        public Task GetMessage()
        {
            IntPtr handle = Handle;
            int bRet;
            while ((bRet = Importer.GetMessage(out var msg, Handle, 0, 0)) != 0)
            {
                switch (bRet)
                {
                    case -1:
                        Console.WriteLine("Error");
                        CancellationToken token = new CancellationToken(true);
                        return Task.FromCanceled(token);
                    default:
                        Importer.TranslateMessage(ref msg);
                        Importer.DispatchMessage(ref msg);
                        break;
                }
            }
            return Task.FromResult(true);
        }
    }
}

Run this in your main Method in the main thread and your menu/gui in secondary thread

 Task.Run(ShowMenu);

 _customWindow.GetMessage();

Importer is custom class containing c++ marshalled functions to create/handle window, look the up by the name as they are the same. All CAPS class/struct are windows/c++ api structs, those can be found on official msdn.