Strange decompilation when using gcc with different optimization

Question

I am running on linux 5.4.18-1-MANJARO with gcc version 9.2.0 (GCC)

Filename:a.c

#include<stdio.h>

int
main(void) {
    int a;

    scanf("%d", &a);

    if (a < 5 || a > 6)
        puts("fail");
    else
        puts("succeed");
}

Then I run:

gcc a.c -O0 -o a.out
gcc a.c -O1 -o b.out

I decompile a.out with r2, and I got this

undefined8 main(void)
{
    undefined8 uVar1;
    int64_t in_FS_OFFSET;
    int32_t var_ch;
    int64_t canary;

    canary = *(int64_t *)(in_FS_OFFSET + 0x28);
    sym.imp.__isoc99_scanf(0x2004, &var_ch);
    if ((var_ch < 5) || (6 < var_ch)) {
        sym.imp.puts("fail");
    } else {
        sym.imp.puts("succeed");
    }
    uVar1 = 0;
    if (canary != *(int64_t *)(in_FS_OFFSET + 0x28)) {
        uVar1 = sym.imp.__stack_chk_fail();
    }
    return uVar1;
}

This is what I expected.

But when I decompile b.out, I got this

undefined8 main(void)
{
    undefined8 uVar1;
    undefined8 extraout_RDX;
    int64_t iVar2;
    int32_t *piVar3;
    uint32_t uVar4;
    int64_t in_FS_OFFSET;
    int32_t iStack20;
    int64_t iStack16;

    iStack16 = *(int64_t *)(in_FS_OFFSET + 0x28);
    piVar3 = &iStack20;
    sym.imp.__isoc99_scanf(0x2004, piVar3);
    if (iStack20 - 5U < 2) {
        uVar4 = 0x200c;
        sym.imp.puts("succeed");
    } else {
        uVar4 = 0x2007;
        sym.imp.puts("fail");
    }
    if (iStack16 != *(int64_t *)(in_FS_OFFSET + 0x28)) {
        sym.imp.__stack_chk_fail();
        sym._init();
        iVar2 = 0;
        do {
            uVar1 = (**(code **)(segment.LOAD3 + iVar2 * 8))((uint64_t)uVar4, piVar3, extraout_RDX);
            iVar2 = iVar2 + 1;
        } while (iVar2 != 1);
        return uVar1;
    }
    return 0;
}

which seems only check if iStack20 < 7, and b.out runs fine.

I don't understand how it works.

Answer 1

Here, an integer underflow is exploited. The operation

iStack20 - 5U

is performed unsigned due to the implicit conversion rules when the two operands differ in type (C standard 6.3.1.8(1)):

Otherwise [signedness differs], if the operand that has unsigned integer type has rank greater or equal to the rank of the type of the other operand, then the operand with signed integer type is converted to the type of the operand with unsigned integer type.

So if iStack20 is < 5, then the result will be very large (somewhere near UINT_MAX) , hence the comparison

iStack20 - 5U < 2

will be false. If iStack20 is > 6, then the result will be greater than 2 anyway. So the program logic is preserved.