WinDBG under VMware Workstation Pro 16.2.3 zeros x87 FPUInstructionPointer when single-stepping


I am learning exploit development and one of the topics is on writing shellcode.

Typically, msfvenom would do the job well with shikata ga nai encoding. The shellcode generated will also usually contains FPU instructions to obtain EIP to make it PIC.

The FPU instructions that I mentioned are:

dadc            fcmovu st, st(4)
d97424f4        fnstenv [esp-0Ch]
5d              pop ebp


Microsoft Windows Version 1709 (Build 16299.2166) x86
Used WinDbg: 10.0.22621.755 X86 for debugging

Expected outcome

When i was using the lab's VM, i was able to get the FPULastInstructionOpcode, where esp would point to the stack address containing the EIP value where fcmovu instruction was executed, hence the pop ebp would place the eip value to ebp.

Received problem

Before i execute the fcmovu instruction, this is how my stack looked like:

01e0744c 41 41 41 41 41 41 41 41 83 0c 09 10 43 43 43 43  AAAAAAAA....CCCC
01e0745c 90 90 90 90 90 90 90 90 90 90 be 02 61 13 10 da  ............a...
01e0746c dc d9 74 24 f4 5d 29 c9 b1 52 83 c5 04 31 75 0e  ..t$.])..R...1u.
01e0747c 03 77 6f f1 e5 8b 87 77 05 73 58 18 8f 96 69 18  .wo....w.sX...i.

After I execute the fnstenv instruction:

01e0744c 41 41 41 41 7f 02 ff ff 41 00 ff ff fe ff ff ff  AAAA....A.......
01e0745c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ff ff  ................
01e0746c dc d9 74 24 f4 5d 29 c9 b1 52 83 c5 04 31 75 0e  ..t$.])..R...1u.
01e0747c 03 77 6f f1 e5 8b 87 77 05 73 58 18 8f 96 69 18  .wo....w.sX...i.

my ESP is pointing at 0x1e0745c, where the EIP value is expected to be in. However, i received NULL value instead of the EIP value.


So what I would like to ask from the community here is why am I getting NULL value? What am i doing wrongly?

I have read the x86 documentation and know that fcmovu is not the problem here.

This is exactly the same shellcode that i have used in the lab and i was able to receive the EIP value as expected.

I have also read from another post made in StackExchange here but the reply didn't feel like it answered the OP's question.

Mitigated solution

With the help of Peter Cordes, I have found that WinDbg under VMWare Guest OS returns null values from FPULastInstructionOpcode when single stepping through x87 FPU instructions. This was an unexpected outcome and was mitigated by setting a breakpoint after the FPU instructions and running the shellcode. Not sure why the debugger in the VMWare clobbers the results like this, but some things just don't really need answers (or rather, probably too much work to investigate something that could be trivially mitigated).


  • This is a bug in your VM or your debugger. If you can figure out which, report it to them. Single-stepping should have the same result as letting the instructions run until you hit a breakpoint (which you confirm is working; thanks for testing that.)

    This works correctly in GDB on Linux (on my i7-6700k CPU directly, no VM). Even single-stepping fcmov then fnstenv, it stores the address of the fcmov as the FPUInstructionPointer in the FP environment, not 0x00000000. (That's what pop ebp loads.)

    The bytes written to the stack on my system match yours for FPU control word1, status word, and tag word, so your FPU was in the same state. That shouldn't matter anyway; setting FPUInstructionPointer to point at the last non-control FP instruction executed always happens, not just on FPU exceptions, according to vol.1 of Intel's x86 manuals (section 8.1.8 x87 FPU Instruction and Data (Operand) Pointers).

    I don't think there's any reason your AMD CPU would clear the x87 FPUInstructionPointer field while taking a single-step debug exception, so it's very likely a software bug, not a hardware difference between your AMD and my Intel. The fact that it works when not stopping between fcmov and fnstenv confirms that AMD hardware does set this field the way Intel documents. I guess it's possible (but unlikely) that AMD's frstor or xrstor doesn't properly restore that field, so make sure you include HW details when reporting this to the VM or debugger developers.

    Footnote 1: Actually there's one difference: yours has control word = 0x027f (precision control = 0b10 = 53-bit mantissa like 32-bit MSVC sets), vs. mine having 0x037f (precision control = 0b11 = 64-bit mantissa, like the finit state.)

    But that doesn't affect what happens with the masked x87 stack underflow exception caused by fcmov when all the x87 registers were in "unwritten" state. The tag word confirms that was the case, so fcmov did fault and write a NaN to st0, leaving the tag field for FP register #0 = 0b10 with the rest empty (tag = 0b11). And the status word confirms C1 = 0 as fcmov sets on stack underflow.

    Not that that matters; FPUInstructionPointer is set to the last non-control FPU instruction whether it faults or not; I had that wrong when commenting on the question and suggesting that a non-faulting fcmov might account for the null pointer.