I'm encountering a situation where a PInvoke call to CloseHandle is throwing an SEHException in a .NET 4 application when run under a debugger. Unlike others who have encountered similar issues migrating from 3.5 to 4, I'm not particularly bothered by the behaviour, and have already located the problem (a third party library calling CloseHandle twice on the same handle). However, I am perplexed as to why this behaviour doesn't happen in a .NET 3.5 application.
The following small but complete example demonstrates the behaviour I'm experiencing (tested on both XP SP3 and Win 7 x64, always compiled as x86):
class Program
{
static void Main(string[] args)
{
try
{
var hFileMapping = CreateFileMapping(new IntPtr(-1), IntPtr.Zero, 0x04 /* read write */, 0, 0x1000, null);
CloseHandle(hFileMapping);
CloseHandle(hFileMapping);
Console.WriteLine("No exception");
}
catch (Exception ex)
{
Console.WriteLine(ex);
}
Console.ReadKey();
}
[DllImport("kernel32", SetLastError = true)]
static extern IntPtr CreateFileMapping(IntPtr hFile, IntPtr lpAttributes, int flProtect, int dwMaximumSizeHigh, int dwMaximumSizeLow, string lpName);
[DllImport("kernel32", SetLastError = true)]
static extern bool CloseHandle(IntPtr handle);
}
When run as a .NET 4 application, an SEHException is thrown at the second CloseHandle. As per the documentation for CloseHandle, this is expected behaviour:
If the application is running under a debugger, the function will throw an exception if it receives either a handle value that is not valid or a pseudo-handle value. This can happen if you close a handle twice, or if you call CloseHandle on a handle returned by the FindFirstFile function instead of calling the FindClose function.
However, when compiled as a .NET 3.5 application (or CLR 2.0), no exception is thrown at the second CloseHandle call, and the message "No exception" is printed.
According to this article, the updated CLR released for .NET 4 has some different default behaviour with low-level exceptions that have potential to corrupt the process state. However, as far as I can understand from that article there is nothing mentioned of previous CLR behaviour that would cause the exception to be completely ignored.
Why does a .NET 3.5 (or CLR 2.0) application not exhibit the documented behaviour of CloseHandle that is present in .NET 4?
Windows only generates the SEH exception when it sees that a debugger is attached. A native debugger. There was a change in the .NET 4 managed debugger that now makes Windows see such a debugger. I can't find any decent links that document the exact details of this new behavior, sorry.
Edit: I found a decent one. Bullet 8 at the bottom of this blog post:
Under the hood we're built on the native debugging pipeline In v2-compat mode, ICD continues to own the pipeline to the target process (since that was the V2 model), but that pipeline is no longer a collection of shared IPC objects with the target process, but is instead the same pipeline a native debugger uses. Specifically, we attach to a process by calling kernel32!DebugActiveProcess, and get our managed events (things that result in calls to ICorDebugManagedCallback) using kernel32!WaitForDebugEvent. This also means that kernel32!IsDebuggerPresent now returns true when doing managed-only debugging. This also has the nice side-effect of avoiding the problem with doing managed-only debugging when a kernel debugger is enabled (the OS assumes any breakpoint instructions that occur when a debugger isn't attached should cause a break in the kernel debugger).
This is not just a cosmetic fix btw, albeit that handle recycle attacks is something that keeps Microsoft employees awake at night. The .NET 4 version of the CLR is built with the CRT version that checks for buffer overflows. When one is detected, the CRT immediately terminates the program. No AppDomain.UnhandledException, it is an immediate crash to the desktop. This code however does the same thing that Windows does, check for a native debugger and generate a breakpoint when one is attached. So it was pretty important that the managed debugger started looking like a native one, the only way to really diagnose the crash.