PInvoke only works in 64 bit, when returning a struct

Question

I have the following C++ code, that compiles to a dll:

typedef struct _RGB {
    unsigned char R, G, B;
} RGB;

extern "C" __declspec(dllexport) RGB __stdcall TestMethod1() {
    RGB rgb{1,2,3};
    return rgb;
}

and am calling it in C# using:

static void Main(string[] args)
{
    var res = TestMethod1();
}

[DllImport(@"D:\Develop\res\VSProjects\ConsoleApp1\Debug\Dll1.dll", CallingConvention = CallingConvention.StdCall)]
static extern RGB TestMethod1();

[StructLayout(LayoutKind.Sequential)]
struct RGB { public byte R, G, B; }

When running it as x86, after building the dll as x86, I get an error Attempted to read or write protected memory.. In x64 it works fine.

When I use a managed/native debugger, I see it's crashing on return rgb;.

When changing the return type to a long (int in C#) it works fine even as x86.

The RGB struct is blittable so why am I getting this issue?

Answer 1

Don't use struct for "complex" return types, prefer something like this:

C++:

extern "C" __declspec(dllexport) void __stdcall TestMethod2(RGB *prgb) {
    prgb->R = 1;
    prgb->G = 2;
    prgb->B = 3;
}

C#:

[DllImport(@"D:\smo\source\repos\ConsoleApplication4\Debug\Dll1.dll")]
static extern void TestMethod2(ref RGB rgb);

static void Main(string[] args)
{
    var rgb = new RGB();
    TestMethod2(ref rgb);
}

Note in your particular case, it fails because the structure size is 3, so you can make it work if you change the structure like this for example:

C++:

typedef struct _RGB {
    unsigned char R, G, B, A;
} RGB;

C#

[StructLayout(LayoutKind.Sequential)]
struct RGB { public byte R, G, B, A; }

With this definition, the size will be 4, so the C++ compiler will generate a code that will return an int32 value instead of returning - probably - a reference to some internal memory that will be gone by the time execution reaches the .NET side. This is pure luck (or hack) and depends on the C++ compiler I guess.