Search code examples
c#.netcexceptionpinvoke

Dealing with a 3rd party library who's function call causes heap corruptions when P/Invoking

I have created a library wrapper which invokes FlexNet Publisher. It is written in C. I am trying to P/Invoke it:

[DllImport("lmgr11.dll")]
public static extern void free_job();

Its implementation is fairly simple:

void WINAPI free_job()
{
    if (jobPtr != NULL)
        lc_free_job(jobPtr);
    jobPtr = NULL;
}

The documentation says that lc_free_job should free a job as well as all resources. Calling lc_free_job works just fine from native code (I made an ATL object expose a wrapper to it through a COM object and I can consume this method all day long from a Visual C++ console application, so I know it must work).

However, from C#, when I try to P/Invoke this method, I get the following error which crashes my application:

Unhandled exception at 0x00007FFA39358283 (ntdll.dll) in CerberusTestHarness.exe: 0xC0000374: A heap has been corrupted (parameters: 0x00007FFA393AF6B0).

Why is this? Can I catch it or circumvent it in any way? I can't seem to catch the exception if I wrap the call in a try-catch because its not throwing a .NET exception. Note that I don't have the source code for lc_free_job so I can't inspect it or view its source code, unfortunately. Its written in C.

Solution

  • The issue was due to my C# P/Invoke code from a previous invocation. Hopefully this helps anyone else who comes across the same issue.

    I had defined this P/Invoke to get an error string back from a function which returns char *:

    [DllImport("lmgr11.dll")]
public static extern string errstring();

    I was calling it every time something failed in my 3rd party library as above. This is not the correct way to P/Invoke a method which returns char *. Otherwise, when freeing the error from the native side, it will cause a heap corruption due to the way this string was marshaled.

    It needed to be defined like this:

    [DllImport("lmgr11.dll")]
public static extern IntPtr errstring();

    And called as follows:

    var errorMessage = Marshal.PtrToStringAnsi(errstring());