c# Pinvoke with StringBuilder returns different results for the same method

So I noticed something very odd that I failed to fix. I'm sure it's just a stupid encoding error, but trying to encode the results differently didn't help me either.

The Problem:

When I Pinvoke the function PathYetAnotherMakeUniqueName from the Shell32.dll via DllImport

[DllImport("shell32.dll", EntryPoint = "PathYetAnotherMakeUniqueName", CharSet = CharSet.Unicode)]
internal static extern bool PathYetAnotherMakeUniqueName2(
                System.Text.StringBuilder pszUniqueName,
                string pszPath,
                string pszShort,
                string pszFileSpec);

[...snip...]

System.Text.StringBuilder pszUniqueName = new System.Text.StringBuilder();
PathYetAnotherMakeUniqueName2(pszUniqueName, "Foo", "Bar", "Test");
Console.WriteLine(pszUniqueName); // Output : Foo\Bar

Or via the LoadLibrary approach

internal delegate bool dPathYetAnotherMakeUniqueName([MarshalAs(UnmanagedType.LPWStr)]System.Text.StringBuilder pszUniqueName, string pszPath, string pszShort, string pszFileSpec);
static dPathYetAnotherMakeUniqueName PathYetAnotherMakeUniqueName;

[...snip...]

hCurModule = LoadLibrary("shell32.dll");
IntPtr pFunction = GetProcAddress(hCurModule, "PathYetAnotherMakeUniqueName");
PathYetAnotherMakeUniqueName = (dPathYetAnotherMakeUniqueName)Marshal.GetDelegateForFunctionPointer(pFunction, 
            typeof(dPathYetAnotherMakeUniqueName));

System.Text.StringBuilder pszUniqueName = new System.Text.StringBuilder ();
PathYetAnotherMakeUniqueName(pszUniqueName, "Foo", "Bar", "Test");
Console.WriteLine(pszUniqueName); // output: "?o\?r???o

As you can see I get two only vaguely similar outputs. I tried changing the MarshalAs attribute or encoding the StringBuilders' contents in ASCII, Unicode or UTF7,8,32 etc., but the output did not change. Without using MarshalAs (which seems to work for some other methods that require the StringBuilder) I get a stack imbalance exception.

Did anybody encounter this problem before?

Solution

It seems that your input strings are being marshalled as LPStr. The default marshalling for strings is UmanagedType.LPTStr, which is platform dependent and is resolved to either LPStr or LPWStr. On your platform it is being resolved to LPStr, I'm assuming it is because the default CharSet is CharSet.ANSI, and you do not specify a different one for the manually loaded function (would welcome authoritative confirmation).

Explicitly specifying the input parameters as [MarshalAs(UnmanagedType.LPWStr)] solves the problem.

internal delegate bool dPathYetAnotherMakeUniqueName(
    [MarshalAs(UnmanagedType.LPWStr)]System.Text.StringBuilder pszUniqueName, 
    [MarshalAs(UnmanagedType.LPWStr)]string pszPath,
    [MarshalAs(UnmanagedType.LPWStr)]string pszShort, 
    [MarshalAs(UnmanagedType.LPWStr)]string pszFileSpec);

The CharSet = CharSet.Unicode field in the DllImport attribute takes care of this for you, hence the difference.

A more straightforward method is to use the UnmanagedFunctionPointer attribute when declaring your delegate, and specify the proper CharSet. This more closely matches your DllImport declaration.

[UnmanagedFunctionPointer(CallingConvention.Winapi, CharSet=CharSet.Unicode)]
internal delegate bool dPathYetAnotherMakeUniqueName(
    System.Text.StringBuilder pszUniqueName, 
    string pszPath,
    string pszShort,
    string pszFileSpec);

More to the point, it doesn't seem like there is any reason to use GetProcAddress in this case. You already know the name of the dll in advance, you don't need to specify its full path, and you can guarantee that the dll will be found and that the function will be found within in. DLLImport already handles the situation perfectly.