P/Invoke Marshalling an Out void*

I have the following C function definition:

EXPORT CreateWindow(const wchar_t* applicationTitle, void* windowHandle) {
    // Some preconditions & other stuff here

    windowHandle = RENDER_COMPONENT.Window.CreateNew(cstr_to_wstring(applicationTitle));
    return true;
}

The function is called via P/Invoke. The P/Invoke function definition is as follows:

[DllImport(InteropUtils.RUNTIME_DLL, EntryPoint = "CreateWindow", CallingConvention = CallingConvention.Cdecl)]
internal static extern bool _CreateWindow([MarshalAs(UnmanagedType.LPWStr)] string applicationTitle, [Out] out IntPtr windowHandle);

The usage of this function is:

IntPtr windowHandle;
bool windowCreationSuccess = _CreateWindow(Engine.ApplicationTitle, out windowHandle);

My problem is that windowHandle is always equal to IntPtr.Zero in the C# code, which I guess means that it isn't being 'copied' from the C++ to C# side somehow. It is being set in the C function (I looked with the debugger)- it is actually a HWND.

P/Invoke never ceases to confuse me - I'm sure I've done something wrong with copying a struct rather than a reference to it or something, but I can't quite see what/where.

Solution

C uses pass by value exclusively. This means that your assignment to windowHandle can never be seen by the caller. That is so for any caller, not just a managed pinvoke.

You need to change the C function to receive the address of the window handle variable. Like this:

EXPORT CreateWindow(const wchar_t* applicationTitle, HWND* windowHandle) {
    // Some preconditions & other stuff here
    *windowHandle = RENDER_COMPONENT.Window.CreateNew(cstr_to_wstring(applicationTitle));
    return true;
}

The C# code in the question then matches.

However your C code is odd. Why do you return true unconditionally? Why bother returning a bool if it is always true. Frankly it would be cleaner to return the HWND.