C++ dll and C# call

Question

I have a function made in C++ that calls a COM interface's function Its signature:

BOOL func(LPWSTR strIn, __out LPWSTR strOut)
{
  //initcom
  //do something
  // release pointers
}

In C#:

[DllImport("funcdll.dll")]
static extern bool func(String strIn, ref String strOut);

// use it

for(int i=0;i<10;i++)
{
   if(func(strin, strout))
   {
      //do something with strout
   }
}

I have tested my dll in a C++ console application, it works, but in C# it crashes with an unknown error.

Answer 1

You've got three problems that I can see.

1. The calling conventions don't match. Your C++ code is cdecl and your C# code is stdcall.
2. The C++ code uses wide strings, but the C# code marshals ANSI strings.
3. The second parameter doesn't match. Your C# code assumes that the C++ code returns a new pointer to a C string which the C# code then deallocates with the COM allocator. Your C++ code doesn't do this.

Now, dealing with these in more detail.

Calling conventions

This is pretty easy to fix. Simple change the C++ code to stdcall, or the C# code to cdecl. But don't do both. I'd change the C# code:

[DllImport("funcdll.dll"), CallingConvention=CallingConvention.Cdecl]

Unicode/ANSI strings

I presume you are wanting to use Unicode strings since you have explicitly selected them in the C++ code. But P/invoke defaults to marshalling ANSI strings. You can change this again in the DllImport like so:

[DllImport("funcdll.dll"), CallingConvention=CallingConvention.Cdecl, 
    CharSet=CharSet.Unicode]

Returning a string from C++ to C#

Your current C++ function declaration is so:

BOOL func(LPWSTR strIn, __out LPWSTR strOut)

The __out decorator has no real effect, other than documenting that you want to modify the buffer pointed to by strOut and have those modifications returned to the caller.

Your C# declaration is:

static extern bool func(String strIn, ref String strOut);

Now, ref String strOut simply does not match. A ref string parameter matches this in C++:

BOOL func(LPWSTR strIn, LPWSTR *strOut)

In other words the C# code is expecting you to return a new pointer. In fact it will then proceed to deallocate the buffer you returned in strOut by calling CoTaskMemFree. I'm confident that's not what you want.

Your original C++ code can only return a string to the C# code by modifying the buffer that was passed to it. That code would look like this:

BOOL func(LPWSTR strIn, __out LPWSTR strOut)
{
    ...
    wcscpy(strOut, L"the returned string");
    ...
}

If this is what you want then you should allocate a sufficient buffer in C# in a StringBuilder object.

[DllImport("funcdll.dll"), CallingConvention=CallingConvention.Cdecl, 
    CharSet=CharSet.Unicode]
static extern bool func(string strIn, StringBuilder strOut);
...
StringBuilder strOutBuffer = new StringBuilder(128);
bool res = func("input string", strOutBuffer);
string strOut = StringBuilder.ToString();

If you simply cannot decide in the C# code how big a buffer you need then your best bet is to use a BSTR to marshal strOut. See this answer for details.