Search code examples
c#.netpinvoke

C# PInvoke: external C++ api works well when I press "F5" to run in VS but can not work outside

When I press "F5" to run my C# application in Visual Studio 2022, MDA suggests PInvokeStackImbalance, but if I ignore MDA and press F5 to continue, it works well, C++ api returns 1.

But if I run my application directly in Windows, there is no exception appears and api returns 0.

Platform is set to Release - x86, and I confirmed the folder is correct. My C# project is .NET Framework 4.7.2, winform application.

Hope someone can help me to fix the PInvokeStackImbalance issue - I know fix it may make my application works outside VS.
And, more important, tell me why "F5" in Visual Studio can work but can not work outside.

C++ Api definition (.h):

UPGRADE_API BOOL _stdcall WriteData(USHORT nID, BYTE dest, PBYTE pDataBuffer, USHORT nBufferSize, DWORD dwLayer=0);

//...

#define UPGRADE_API __declspec(dllimport)

My C# codes:

[DllImport("Upgrade.dll", EntryPoint = "WriteData", CallingConvention = CallingConvention.StdCall)]
public static extern int WriteData(int id, byte dest, char[] data, int len);

//...

    var resource = "Test";
    var array = resource.ToCharArray();
    var r = WriteData(33, 0, array, array.Length);
    if (r == 0)
    {
        MessageBox.Show("Failed");
        return;
    }
    MessageBox.Show($"Ok: {resource}");

C++ api is a third-party dll and I only have the .h file so I can not debug with c++ code directly.

I tried to build my C# application with x64 and c++ api crashed. And I built debug, same to release.

I checked Start options in VS, it is empty.

I tested many data types of api parameters, such as uint/ushort/short id, StringBuilder/string data, int/ushort len. Some types are wrong and even "F5" in VS also returns 0. But all of these types, the PInvokeStackImbalance always occurs.

Solution

  • The C++ prototype of WriteData, there are 5 parameters. One of them has a default value, but you still need to pass it: a parameter with a default value is always passed to the called function, you can omit it from a call but the way that is handled is by the compiler inserting the value that was set as the default value of that parameter. You cannot omit it from the dll import because then that parameter really is not passed, resulting (for an stdcall function) in pushing less data to the stack than the called function deallocates when it returns (stdcall functions are responsible for deallocating their own parameters). Hence stack imbalance.

    Additionally PBYTE pDataBuffer suggests that you should pass a byte[] in C#, perhaps not, but that type does not suggest that WriteData works with 16-bit characters (as char is in C#).

    The different types you've tried would for the id and len parameters would not make a significant difference, when they're pushed to the stack in x86 they'd all be pushed as dword.