Search code examples
c#commarshallingvisual-studio-2022visual-studio-extensions

Prevent the CLR from freeing memory of a marshalled double char pointer?


I am building a Visual Studio 2022 extension in C# and am currently playing around with the VS API. I am trying to call IVsObjectList2.GetText(), which is a COM interface. However, VS is crashing while attempting to do so. I suspect that the marshalling definition is wrong in the VS API.

The documentation states that the 3rd argument to GetText() (the string) should NOT be freed by the caller. I.e. the native interface is defined as

HRESULT IVsObjectList2::GetText(  
   [in] ULONG Index,   
   [in] VSTREETEXTOPTIONS tto,   
   [out] const WCHAR **ppszText  
);

and ppszText must not be freed by the caller of GetText(). However, in the decompilation of the C# code I can see that GetText() is defined as

[MethodImpl(MethodImplOptions.PreserveSig | MethodImplOptions.InternalCall)]
int GetText(
  [In][ComAliasName("Microsoft.VisualStudio.OLE.Interop.ULONG")] uint index,
  [In][ComAliasName("Microsoft.VisualStudio.Shell.Interop.VSTREETEXTOPTIONS")] VSTREETEXTOPTIONS tto,
  [MarshalAs(UnmanagedType.LPWStr)] out string ppszText);

From what I understand, the default marshalling behavior is causing the CLR to free the string argument ppszText automatically. Is my understanding correct?

If I am correct, since this is then at odds with the API definition, can I somehow hack around the issue? I mean can I somehow correct the marshalling definition of GetText() in my own code? (I think the third argument should be an IntPtr instead?) For example, if this were C++, I would maybe do a "reinterpret_cast" of IVsObjectList2 to an own definition that is then correct; is something like this possible in C#?

Calling C# code:

IVsObjectManager2 objectManager2 = MyExtensionPackage.GetGlobalService(typeof(SVsObjectManager)) as IVsObjectManager2;
if (objectManager2.FindLibrary(new Guid(BrowseLibraryGuids80.VC), out IVsLibrary2 lib) == VSConstants.S_OK) {
  var criteria = new VSOBSEARCHCRITERIA2();
  if (lib.GetList2(
        (uint)_LIB_LISTTYPE.LLT_CLASSES, 
        (uint)_LIB_LISTFLAGS.LLF_DONTUPDATELIST, 
        new[] { criteria }, 
        out IVsObjectList2 list) == VSConstants.S_OK
      && list != null) {
    if (list.GetItemCount(out uint count) == VSConstants.S_OK && count > 0) {
      for (uint idx = 0; idx < count; ++idx) {
        // The following crashes eventually.
        list.GetText(idx, VSTREETEXTOPTIONS.TTO_PREFIX, out string prefix);
        // Use "prefix"
      }
    }
  }
}

The list.GetText() call doesn't crash immediately, but eventually does so because of some heap corruption with the following callstack:

ntdll.dll!RtlpBreakPointHeap()
ntdll.dll!RtlpValidateHeapEntry()
ntdll.dll!RtlValidateHeap()
AcLayers.dll!NS_FaultTolerantHeap::APIHook_RtlFreeHeap(void *,unsigned long,void *)
mscorlib.ni.dll!00007ffe31eb071f()
Microsoft.VisualStudio.Interop.ni.dll!00007ffe2d8bfc53()
Microsoft.VisualStudio.Interop.ni.dll!00007ffe2d8bfbe7()
[Managed to Native Transition]
My own code in C#, line with GetText()

Solution

  • I mean can I somehow correct the marshalling definition of GetText() in my own code?

    One thing you can generally do in .NET is you can define your own copy of IVsObjectList2 interface with the same GUID attribute, and same method ordering, and then cast the object to your interface. That would then allow you to redefine the GetText method with a more appropriate signature. Unfortunately this can be a pain, since you need to be careful to do things like keep the interface members in the same order as the underlying IDL.