Any experts in .NET PInvoke marshalling know why using an int to hold a 64-bit pointer value works most of the time in a 64-bit app? Embarrassingly we found some code that wasn't properly ported to 64-bit. We have a native DLL that returns a "struct foo *" and we've been holding that value in a C# int. We later return that to the native DLL to close the "handle". Amazingly it has been working and passing our 64-bit tests.
When I look at the pointer value returned from the native code it always stay within the range of an int. What is odd about this is a customer found the issue by running their 64-bit app in the VS debugger. The app crashes with an AccessViolationException. We've subsequently found that it only fails when debugged using vshost.exe. Something about the app running in vshost.exe excites the problem. If you uncheck "Enabled the Visual Studio hosting process" or enable native debugging, the application doesn't crash. This is under .NET 4.0.
I'm a bit mystified that this works at all and why would the vshost.exe process excite the crash. Any theories as to why vshost.exe would excite this bug? I'm also a bit disappointed the VS managed debug assistants didn't catch this issue.
A 64-bit process can use addresses < 4GB as well as > 4GB. There is no particular guarantee either way, it is up to the operating system's memory manager. The hosting process is not much of a lead, the Windows version is a stronger one. Windows 8 tends to allocate at high addresses more frequently than 7 for example.
An MDA cannot catch this, it has no idea that the native code actually expects a 64-bit value. The 64-bit calling convention helps to hide this, the first 4 arguments are passed through registers so there is no stack imbalance. And the upper 32-bits of the register are very often 0 by accident. If it is passed through a structure or object then you tend to get lucky (well, unlucky) because of structure packing. The next field can get aligned and leave enough space to still have the layout that the native code expects. These padding bytes are 0 due to the way the CLR initializes memory.