Byte array is not recognised when calling external C++ method

Question

There is C++ api:

typedef struct
{
    BYTE    bCommandCode;
    BYTE    bParameterCode;

    struct
    {
        DWORD   dwSize;
        LPBYTE  lpbBody;
    }
    Data;
}
COMMAND;

And a function:

DLL_API DWORD WINAPI ExecuteCommand( LPCSTR, CONST COMMAND, CONST DWORD, LPREPLY);

And my C# equivalent code:

public struct Data
{
    public int dwSize;
    public byte[] lpbBody;
}

public struct Command
{
    public byte bCommandCode;
    public byte bParameterCode;

    public Data Data;
}

[DllImport(@"api.dll", CallingConvention = CallingConvention.Winapi)]
public static extern int ExecuteCommand(string port, Command command, int timeout, ref Reply reply);

Reply struct is not necessary here.

I call ExecuteCommand:

Command command = new Command();
command.bCommandCode = 0x10;
command.bParameterCode = 0x10;

byte[] bData = { 0xff, 0xff };
command.Data.dwSize = bData.Length;
command.Data.lpbBody = bData;

Reply reply = new Reply();
var result = ExecuteCommand("COM1", command, 5000, ref reply);

When I see logs from C++ dll I see that by byte[] bData is not correctly recognised at all. What I am doing wrong? Maybe this definition is not correct: public byte[] lpbBody? How can I pass array as LPBYTE in the struct to the C++ method?

Answer 1

When you allocate a managed object, (such as the byte array you are having issues with) it is mapped to a certain address in the managed heap, which, in turn, is mapped to a certain unmanaged memory address. The mapping between managed and unmanaged addresses can change when the GC operates, since it de-fragments the unmanaged memory space assigned to it by moving unmanaged memory chunks around.

When you invoke an unmanaged API with a byte[] as a reference, the marshaling process basically passes the unmanaged address of the byte array object to the native API. Thus, it is quite possible that the memory address of the byte array no longer points to what you expect it to when you attempt to use it, due to the aforementioned de-fragmentation.

I sincerely believe this is what you're experiencing.
Luckily, this issue can be easily resolved:

GCHandle pinned = GCHandle.Alloc(bData, GCHandleType.Pinned);
IntPtr arrPtr = pinned.AddrOfPinnedObject();

The first line tells the GC to not fiddle with the Managed -> Unmanaged mapping for this object. The second speaks for itself. All you have to do now is change the 'Data' struct at the C# side to hold an IntPtr instead of byte[]
(no need to change the C++ side).

public struct Data
{
    public int dwSize;
    public IntPtr lpbBody;
}

Be sure to call the GCHandle.Free() method when done with the GCHandle object.

I do hope you are marking your marshaled types with the MarshalAsAttribute class and you simply omitted them in the example.