Search code examples
assemblyx86x86-64integer-division

Catching / disabling SIGFPE exceptions from assembly

I get a SIGFPE exception when dividing by 0 in x86 assembly using idiv. How would I disable it from assembly? Do I need a syscall or can it be done directly in x86?

Reproduction:

test.asm

default rel
global WinMain

section .data 

section .text
WinMain:
    mov rcx, 0
    mov rdx, 0
    idiv rcx

Commands:

nasm -f win64 test.asm
gcc test.obj
gdb a
run

Solution

  • TL:DR: you can't make the hardware not fault with a #DE exception, and it's non-trivial to resume execution after skipping the faulting instruction in a signal handler to "repair" the situation. (You'd also want to set RDX and RAX to some special values like -1, 0, or a noticeable value like 0xcccccccccccccccc. Or depending on what might make the rest of your program silently do something useful(?) instead of crash again a different way, perhaps +1.)

    For actual floating-point exceptions, the x87 FPU and SSE (via MXCSR) default to having exceptions masks, so you get NaN in the destination on invalid ops like division by zero, no hardware exception. POSIX requires that if there's a fault for any arithmetic operation and the OS delivers a signal about it, that signal must be SIGFPE.

    The best you could do is catch SIGFPE with a signal handler (sigaction(2) on POSIX OSes like Linux). Strange that you're talking about SIGFPE for a program with a WinMain entry point, unless perhaps Windows SEH (structured exception handling) also uses the same SIGFPE name?

    Ignoring it with SIG_IGN is undefined according to POSIX, and in practice would result in an infinite loop if you ignore the signal or have a signal-handler return without doing anything. The hardware #DE fault has a return address pointing at the faulting instruction (same as for page faults), both so the handler can see which instruction it is, and so repairing the situation somehow (like a #PF handler normally does) will re-run the instruction to hopefully not fault on the next try.

    So your handler should check if si_code == FPE_INTDIV. If so, it could try to advance the program counter to past the end of the faulting instruction, to run as if it hadn't executed at all, leaving FLAGS and RDX:RAX unmodified. (Or your signal handler could set them to some consistent values.)

    A thread's registers are accessible and modifyable by a Linux signal handler that runs in it. Getting fault address that generated a UNIX signal covers the ucontext_t stuff as well, containing GPRs including RAX and RDX but also RIP, the program counter.

    x86-64 instructions are variable length. You'll have to decode machine code from the address in RIP, skipping prefixes, looking for an opcode byte that's an opcode for idiv or div. If so, advance RIP to after that instruction to skip it. Instruction-length decoding is non-trivial since memory source operands with variable-length addressing modes are possible: you'll have to decode ModRM and the optional SIB enough to figure out the total instruction length. (x86-64 is designed to make this not too painful so hardware can do it efficiently: you don't need to remember the bits from the REX prefix if any while doing instruction-length decoding for the part after the prefixes: *rbp not allowed as SIB base?)

    There's no FLAGS / control register / other hardware setting that will make div and idiv not raise #DE divide exception on divisor = 0 or on the quotient not fitting into the operand-size. e.g. into RAX for 64-bit idiv rcx, like for INT64_MIN / -1 which also faults on x86-64.

    Or for (unsigned long)-1 / 1 with signed division like you're doing by zero-extending into RDX:RAX before signed idiv. Normally use cqo / idiv to sign-extend RAX into RDX:RAX before signed division, or xor edx,edx / div to zero-extend into RDX before unsigned division.
    (When and why do we sign extend and use cdq with mul/div?)

    You can mask FP exception, and the default x87 and SSE (MXCSR) state is for FP exceptions to be masked, so normally the only thing that can raise SIGFPE on an x86-64 system is integer division. (Or perhaps an into instruction, trap if the OF flag is set.)

    Related: