Determining DLLImport arguments and safely calling an unmanaged C function

Question

As a follow-up to my previous question, I finally got the C dll exported and usable in C#, but I'm stuck trying to figure out the proper argument types and calling method.

I've researched here on SO but there doesn't seem to be a pattern to how variable types are assigned.

I see some people suggest a StringBuilder for uchar*, others a byte[], some references to 'unsafe' code, etc. Can anyone recommend a solution based on this specific use-case?

Also note the exception generated as the code stands now, right after the call to the C function.

C function import:

[DllImport("LZFuncs.dll")]
internal static extern long LZDecomp(ref IntPtr outputBuffer, byte[] compressedBuffer, UInt32 compBufferLength); //Originally two uchar*, return is size of uncompressed data.

C function signature:

long LZDecomp(unsigned char *OutputBuffer, unsigned char *CompressedBuffer, unsigned long CompBufferLength)

Used as below:

for (int dataNum = 0; dataNum < _numEntries; dataNum++)
        {
            br.BaseStream.Position = _dataSizes[dataNum]; //Return to start of data.
            if (_compressedFlags[dataNum] == 1)
            {
                _uncompressedSize = br.ReadInt32();
                byte[] compData = br.ReadBytes(_dataSizes[dataNum] - 4);
                IntPtr outData = IntPtr.Zero;
                LZFuncs.LZDecomp(ref outData, compData, Convert.ToUInt32(compData.Length));
                var uncompData = new byte[_uncompressedSize]; //System.ExecutionEngineException was unhandled
                Marshal.Copy(outData, uncompData, 0, Convert.ToInt32(_uncompressedSize));
                BinaryWriter bw = new BinaryWriter(new FileStream("compData" + dataNum + ".txt", FileMode.CreateNew));
                bw.Write(uncompData);
                bw.Close();
            }
            else
            {
                BinaryWriter bw = new BinaryWriter(new FileStream("uncompData" + dataNum + ".txt", FileMode.CreateNew));
                bw.Write(br.ReadBytes(_dataSizes[dataNum]));
                bw.Close();
            }
        }

I assume the C code is clobbering the memory pretty severely if it's breaking the C# caller with a CLR exception like that, but due to how the C code is written, there's absolutely no way to modify it without breaking the functionality, it's effectively a black box. (Written in assembly, for the most part.)

For reference, just a few questions I've read over in an effort to solve this myself:

How do I return a byte array from C++ to C#

Correct way to marshall uchar[] from native dll to byte[] in c#

There have been others but those are the most recent.

Answer 1

OK, that's not too hard to work with. The two buffer parameters are byte arrays. You should declare them as byte[]. The calling convention is Cdecl. Remember that C++ long is only 32 bits wide on Windows, so use C# int rather than C# long since the latter is 64 bits wide.

Declare the function like this:

[DllImport("LZFuncs.dll", CallingConvention = CallingConvention.Cdecl)]
internal static extern int LZDecomp(
    [Out] byte[] outputBuffer, 
    [In] byte[] compressedBuffer, 
    uint compBufferLength
);

You are decompressing compressedBuffer into outputBuffer. You'll need to know how large outputBuffer needs to be (the code in the question shows that you already handle this) and allocate a sufficiently large array. Beyond that I think it's obvious how to call this.

The calling code will this look like this:

_uncompressedSize = br.ReadInt32();
byte[] compData = br.ReadBytes(_dataSizes[dataNum] - 4);
byte[] outData = new byte[_uncompressedSize];
int len = LZFuncs.LZDecomp(outData, compData, (uint)compData.Length);