I'm trying to check if signed addition would overflow. Generally, to check if
int a + int b
would overflow (a and b both positive), I check if
if (a > INT_MAX - b)
But now I want to check if
int a + int b - int c
would overflow. I know that a, b, and c are positive and that b >= c, so I do the following check:
if (a > INT_MAX - b + c)
Now my question is, can compiler rewrite
INT_MAX - b + c to INT_MAX + c - b ?
My concern is, then it will first do INT_MAX + c, which may overflow and can result into undefined behavior.
It is a fallacy to think about what "the compiler" does when reasoning about undefined behaviour. The compiler is transparent. The behaviour is in your code, not in the compiler. You should ask "what does my code INT_MAX - b + c mean? Can it overflow? The answer is never in "the compiler", but in the standard.
The standard only requires that individual operations that appear in your program do not overflow. It never says anything about any rewritten expressions that do not explicitly appear in your program. INT_MAX - b + c is in your program, and is equivalent to (INT_MAX - b) + c. So the requirement is that (INT_MAX - b) doesn't overflow, and then the result added to c doesn't overflow. INT_MAX + c - b does not appear in your program, so you should not be concerned about it.
If the compiler rewrites your expression in any way, it must make sure that the rewritten expression has the same visible behaviour as yours, per the as-if rule. So if it does replace INT_MAX - b + c with INT_MAX + c - b, it must make sure the overflow either doesn't happen or is dealt with transparently (e.g. ignored by the hardware).