Converting float to UInt32 - which expression is more precise

Question

I have a number float x which should be in <0,1> range but it undergo several numerical operations - the result may be slightly outside <0,1>.

I need to convert this result to uint y using entire range of UInt32. Of course, I need to clamp x in the <0,1> range and scale it.

But which order of operations is better?

y = (uint)round(min(max(x, 0.0F), 1.0F) * UInt32.MaxValue)

or

y = (uint)round(min(max(x * UInt32.MaxValue, 0.0F), UInt32.MaxValue)

In another words, it is better to scale first, then clamp OR clamp and then scale? I am not very profound in the IEEE floating point representation, but I believe there is a difference in the order of computation of the above expressions.

Answer 1

Because the multiplication to get from [0.0f .. 1.0f] to [0 .. UInt32.MaxValue] can itself be approximative, the order of operations that most obviously has the property you desire is multiply, then clamp, then round.

The maximum value to clamp to is the float immediately below 2³², that is, 4294967040.0f. Although this number is several units below UInt32.MaxValue, allowing any larger value would mean overflowing the conversion to UInt32.

Either of the lines below should work:

y = (uint)round(min(max(x * 4294967040.0F, 0.0F), 4294967040.0F))

In this first version, you have the option to multiply by UInt32.MaxValue instead. The choice is between having very slightly larger results overall (and thus rounding to 4294967040 a few more values that were close to 1.0f but below it), or only sending to 4294967040 the values 1.0f and above.

---

You can also clamp to [0.0f .. 1.0f] if you do not multiply by too large a number afterwards, so that there is no risk of making the value larger than the largest float that can be converted:

y = (uint)round(min(max(x, 0.0F), 1.0F) * 4294967040.0F)

---

Suggestion for your comment below, about crafting a conversion that goes up to UInt32.MaxValue:

if (x <= 0.0f) y = 0
else if (x < 0.5f) y = (uint) round (x * 4294967296.0F)
else if (x >= 1.0f) y = UInt32.MaxValue
else y = UInt32.MaxValue - (uint) round ((1.0f - x) * 4294967296.0F)

This computation considered as a function from x to y is increasing (including around 0.5f) and it goes up to UInt32.MaxValue. You can re-order the tests according to what you think will be the most likely distribution of values. In particular, assuming that few values are actually below 0.0f or above 1.0f, you can compare to 0.5f first, and then only compare to the bound that is relevant:

if (x < 0.5f)
{
  if (x <= 0.0f) y = ...
  else y = ...
}
else
{
  if (x >= 1.0f) y = ...
  else y = ...
}