I'm doing VMWare detection in a .NET 2.0 C# console project. The detection code is implemented in C as an exported function in a DLL that's called from the C# code using P/Invoke. The C# code is compiled as x86 and as x64 into two separate executables. The C DLL is also compiled for both platforms. Here's the exported function:
PUBLIC Int32 __declspec(nothrow) WINAPI GetVMType ()
{
Int32 nVMWareType = 0;
try
{
if ( !IsInVMWare ( nVMWareType ) )
{
... // allocate memory, write data, etc.
}
}
catch ( ... )
{
nVMWareType = -1;
}
return ( nVMWareType );
} // <-- breakpoint happens here
I have the following code using inline assembly for VMWare detection:
PRIVATE Bool IsInVMWare ( Int32& nType )
{
Bool bResult = false;
Int32 nVersion = -1;
nType = -1;
__try
{
#ifndef _WIN64 // 32-bit detection
__asm
{
push edx
push ecx
push ebx
mov eax, 'VMXh'
mov ebx, 0 // anything but 'VMXh'
mov ecx, 10 // get VMWare version
mov edx, 'VX' // port number
in eax, dx // read port
cmp ebx, 'VMXh' // is it a reply from VMWare?
je lblInVMWare
xor ecx, ecx // not in VMWare - clear return value
lblInVMWare:
mov [nVersion], ecx // vmware product type
pop ebx
pop ecx
pop edx
}
#else
nVersion = GetVMWareVersion (); // 64-bit detection
#endif
nType = nVersion;
if ( nType > 0 )
bResult = true;
}
__except ( EXCEPTION_EXECUTE_HANDLER )
{
bResult = false;
}
return ( bResult );
}
The 64-bit detection code is implemented as GetVMWareVersion() in assembly:
PUBLIC GetVMWareVersion
.CODE
ALIGN 8
GetVMWareVersion PROC
push rbp
mov rbp, rsp
push rdx
push rcx
push rbx
mov rax, 'VMXh'
mov rbx, 0 ; anything but 'VMXh'
mov rcx, 10 ; get VMWare version
mov rdx, 'VX' ; port number
in rax, dx ; read port
cmp ebx, 'VMXh' ; is it a reply from VMWare?
je $0@GetVMWareVersion
xor rax, rax ; not in VMWare - clear return value
jmp $1@GetVMWareVersion
$0@GetVMWareVersion:
mov rax, rcx ; VMWare product type
$1@GetVMWareVersion:
pop rbx
pop rcx
pop rdx
mov rsp, rbp
pop rbp
ret
GetVMWareVersion ENDP
END
The 32-bit detection code runs fine on a non-VM Windows 7. When the 64-bit version runs (same environment), it triggers a DebugBreak() call with the following call stack:
KernelBase.dll!DebugBreak() + 0x2 bytes
[Frames below may be incorrect and/or missing, no symbols loaded for KernelBase.dll]
mscorwks.dll!PreBindAssembly() + 0x9ce69 bytes
mscorwks.dll!PreBindAssembly() + 0x9d28e bytes
mscorwks.dll!CreateApplicationContext() + 0x769d bytes
mscorwks.dll!StrongNameTokenFromPublicKey() + 0x64f8 bytes
mscorwks.dll!StrongNameTokenFromPublicKey() + 0x66ff bytes
mscorwks.dll!CreateApplicationContext() + 0x7f62 bytes
ntdll.dll!vsprintf_s() + 0x12b bytes
ntdll.dll!RtlUnwindEx() + 0x852 bytes
ntdll.dll!KiUserExceptionDispatcher() + 0x2e bytes
mscorwks.dll!IEE() + 0xd285 bytes
cccccccccccccccc()
0000000000d78180()
cccccccccccccccc()
I also have this in the Output window when I debug the C DLL in Visual Studio:
First-chance exception at 0x000007feedbdfad1 (mscorwks.dll) in MyApp.exe:
0xC0000005: Access violation reading location 0xffffffffffffffff.
Every once in a while (not always) I also get this in the Event Log:
.NET Runtime version 2.0.50727.5456 - Fatal Execution Engine Error
(000007FEEDB27916) (80131506)
I'm at a total loss as to why this happens. I did a lot of searching but I couldn't find anything that would seem to apply to this problem. Some sites suggest to switch to .NET 4.0 but that's not an option.
I analyzed all the code in GetVMType () - it does some memory allocation and writes data into dynamically allocated arrays but that code is correct: all memory is released and no memory is overwritten incorrectly.
If I modify the 64-bit assembly code to skip the in instruction, the breakpoint is not triggered. This is not an issue when running as 32-bit code.
The calling C# program has a top-level exception handler and an event handler for UnhandledException event but when this problem happens, the application just quits - none of the handlers are called.
Does anyone have any idea what could be wrong with this setup? I spent hours of debugging and trying to see what's happening but it seems something inside .NET breaks when the P/Invoke call returns after that in instruction executes.
My guess is that your function corrups registers.
Running on real hardware (non-VM) should probably trigger exception at "in rax, dx". If this happens then control is passed to your exception handler, which sets result, but does not restore registers. This behaviour will be fully unexpected by caller. For example, it can save something into EBX/RBX register, then call your asm code, your asm code does "mov RBX, 0", it executes, catches exception, sets result, returns - and then caller suddently realizes that his saved data isn't in EBX/RBX anymore! If there was some pointer stored in EBX/RBX - you're going to crash hard. Anything can happen.
Surely, your asm code saves/restores registers, but this happens only when no exception is raised. I.e. if your code is running on VM. Then your code does its normal execution path, no exceptions are raised, registers will be restored normally. But if there is the exception - your POPs will be skipped, because execution will be passed to exception handler.
If this is true, then your 32-bit code doesn't work either. If it's working - then it happens only by coinsidence. I.e. by pure chance. And that chance is that in case of 32-bit code which have less available registers to use, someone already saves/restores registers before calling into your asm code.
The correct code should probably do PUSH/POPs outside of try/except block, not inside.