Significant precision difference in floating point calculations in Java / C#

Question

I know that similar questions have been asked before, but none of the answers solves my problem.

I have 2 functions:

Java

public static void main(String[] args) {
    double h, z, lat0, n0, eSq;
    z    = 4488055.516;
    lat0 = 0.7853981634671384;
    n0   = 6388838.290122733;
    eSq  = 0.0066943799901975545;
    h    = z / Math.sin(lat0) - n0 * (1 - eSq);
    System.out.println(h);
}

C#

public static void Main (string[] args)
{
    double h, z, lat0, n0, eSq;
    z    = 4488055.516;
    lat0 = 0.7853981634671384;
    n0   = 6388838.290122733;
    eSq  = 0.0066943799901975545;
    h    = z / Math.Sin(lat0) - n0 * (1 - eSq);
    Console.WriteLine(h);
}

or

4488055,516/sin(0,7853981634671384)-6388838,290122733*(1-0,0066943799901975545)

for SpeedCrunch, Maxima and LibreOffice Calc.

Results are:

Java:             1000.0000555226579 (same with and without strictfp)
C#:               1000,00005552359
SpeedCrunch (15): 1000,000055524055155
SpeedCrunch (50): 1000,00005552405515548724762598846216107366705932894830
LibreOffice Calc: 1000,000055523590000
Maxima:           1000.000055523589
Maxima:           1.00000005552391142b3 (bfloat - fpprec:20)

As you can see, Java and C# are different at the 9th decimal place. Others are not so uniform either. This is tested on the same OS and same CPU. Tests are done also on 32-bit and 64-bit systems.

How to solve this kind of problem? I thought that precision should be equal to 15 decimal places.

Answer 1

The cause of not getting 15 digits is the subtraction of two similar numbers. z / Math.sin(lat0) is about 6347068.978968251. n0 * (1 - eSq) is about 6346068.978912728. With 7 digits before the decimal point, a change in the 9th decimal place in the subtraction result corresponds to a change of less than one part in 10^15 in one of those inputs.

The simplest solution to this type of problem is usually to display only the digits that are supported by reliable digits in the inputs. Very few measurements can be done to one part in 10^12, so in this case it is almost certain that the digits that differ due to floating point rounding error would be dropped.

For example, it looks as though your data relates to location. One of the most carefully measured pieces of such data is the height of Mount Everest. The current best estimate, based on high precision GPS measurement, is "29,035 feet, with an error margin of plus or minus 6.5 feet" Encyclopedia Britannica. An error in the 13th significant digit corresponds to an error of less than one thousandth of an inch in measuring the circumference of the earth.

If the rounding error really is significant relative to the result requirements, and to what is realistically achievable given the accuracy of the inputs, then you might need to look at more clever ways of arranging the calculation or at higher precision arithmetic.