I have a 2 dimensional vector V represented with integers, and I subtract a constant C from V's magnitude and get a vector W that has the same direction as V but obviously the magnitude of |V| minus C.
The magnitude requires a square root, and integer square roots are an annoyance. I really don't want to convert to floats for this, and in the past I've managed to manipulate equations to remove square roots (like projecting one vector to another)... But this is a tricky one. So anyway, the basic equation is W = (|V| - C) * (V / |V|), I think. I've tried manipulating it but can't remove a magnitude at some point in it.
Is it even possible to subtract from a vector's magnitude without using a square root?
No – if you shorten (1, 1) by 1, then you get (1 - 1/√2, 1 - 1/√2), and there's no way to write 1 - 1/√2 using only +, -, *, /, and integers.