I'm having precision problems when dividing an long type with a float and ended up messing the calculations in the app. Let say I'm just converting a millisecond to second, since that is a requirement on our end. Below is a sample test case
long startTimeMS = 2529095;
NSTimeInterval expectedStartTime = 2529.095
NSTimeInterval startTime = startTimeMS / 1000.f; //Results to "2529.09497"
double startTimeDouble = startTimeMS / 1000.0 // Results to "2529.0949999999998"
XTCAssert(startTime == expectedStartTime, @"Invalid start time")
Upon dividing a long by float, it loses precision by a 3 thousandth. Similar with long divided by double.
How do I make it so that dividing it would come out the same as the expected start time? I'm guessing rounding up to nearest thousandth in decimal would also be good but the rounding methods I found here isn't reliable either like this
float roundToTwo(float num)
{
return round(100 * num) / 100;
}
Which most likely return the same value
You are misunderstanding how floating-point works. The floating-point types float and double are binary types.
Your decimal fraction 0.095 is equal to 9/100 + 5/1000 - all decimal fractions are sums of 1/(10^n) fractions.
Now consider ordinary fractions, what is the decimal fraction equivalent of 1/3? It's 3/10 + 3/100 + 3/1000 + ... or equivalently 0.333..., there is no exact way to represent it as the same of decimal fractions.
For the types float and double the fractional part are sums of 1/(2^n) fractions. Similarly to the above not all decimal fractions can be precisely written as binary fractions.
In addition to the differences induced by using different bases, decimal vs. binary, computer floating-point is limited in the number of digits in any number while real math is unlimited, and thus computations may not produce exact results.
For the above reasons it is good floating-point practice to never test for exact equality but rather whether the difference of two numbers is less than a small value (which may vary depending on the magnitude of the numbers). In your case you might decide that two numbers are "equal" if their difference is less than, say, 0.1ms.
Now having said all that in the particular example you give, if it is corrected, the numbers are actually the same, but neither is 2529.095:
long startTimeMS = 2529095;
NSTimeInterval expectedStartTime = 2529.095; // Result is approx "2529.0949999999998"
NSTimeInterval startTime = startTimeMS / 1000.0; // Result is approx "2529.0949999999998"
These two numbers will compare equal, but this need not be the case for all values. It would be better to write your assert as:
XTCAssert(fabs(startTime - expectedStartTime) <= 0.0001, @"Invalid start time");
where 0.0001 should be whatever fraction of a second is small enough that you deem the two values "equal".
You really should read up on floating-point arithmetic if you are going to be doing work which requires precision, as you do more operations the differences due to base and limited digits accumulate and there are techniques to minimise and deal with these.
HTH