C# PInvoke pass out unsigned char*

Question

I am trying to create a simple dll in C++ and call it from C# using PInvoke. I want to pass in a byte array, do some manipulation on it, and send back another byte array. I figured I would pass in the frame and size then create an unmanaged unsigned char*. I would pass that back in an out IntPtr and return the size. Then later I would free that memory with another function. Everything is working ok except I cannot get the out IntPtr to work. I always just get 0 back. I created my C++ dll in qt. Here is the code I have so far.

#pragma once

#ifdef TOOLS_EXPORTS
#define TOOLS_API __declspec(dllexport)
#else
#define TOOLS_API __declspec(dllimport)
#endif

class Tools
{
public:
    int TOOLS_API TestFunction(const unsigned char *inData, int inSize, unsigned char *outData);
};

---

#include "Tools.h"

int Tools::TestFunction(const unsigned char *inData, int inSize, unsigned char* outData)
{
    outData = (unsigned char*)malloc(sizeof(unsigned char) * inSize);

    memcpy(outData, inData, sizeof(unsigned char) * inSize);

    return inSize;
}

---

[DllImport("Tools.dll", EntryPoint = "?TestFunction@Tools@@QAEHPBEHPAE@Z", CallingConvention = CallingConvention.StdCall)]
public static extern int TestFunction(byte[] inData, int inSize, out IntPtr outData);

---

IntPtr outData;

int test = TestFunction(data, data.Length, out outData);

Answer 1

You have the final parameter as:

unsigned char* outData

This is a pointer, passed by value. Any modifications to the address are not seen by the caller, because the address is passed by value.

You need to return the address of the memory you allocate to the caller. So you need:

unsigned char** outData

Then in the implementation of the function in your C++ code you replace references to outData with *outData.

By definition sizeof(unsigned char) == 1 and it would be idiomatic to replace sizeof(unsigned char) * inSize with inSize.