While experimenting with code to convert double to text, I am comparing results of my routine to those from standard library. For the number and formatting in the title, my routine returns
0.0039062
and the same result is printed by libgcc printf. However, Microsoft C++ (build tools 2019) says
0.0039063
I believe Microsoft's library is in error, per round to even rule. Am I right?
The C standard says the result must be correctly rounded, but it is not explicit about what the rounding rule is.
Some preliminaries: .00390625 = 2−8 is exactly representable in double if it uses a binary or decimal format, because DECIMAL_DIG (declared in <float.h>) must be at least ten, and this implies the precision is sufficient for exact representation. (Theoretically, a double could have another base, such as three, and then .00390625 would not be exactly representable. This answer does not discuss such cases.)
C 2018 7.21.6.1 7 says a %.5g conversion with this number will use the f style. For the f style, it says “… The value is rounded to the appropriate number of digits.”
C 2018 7.21.6.1 13 says “For e, E, f, F, g, and G conversions, if the number of significant decimal digits is at most DECIMAL_DIG, then the result should be correctly rounded…” Five is of course less than ten, so the result should be correctly rounded.
“Correctly rounded” is defined in 3.9 as “representation in the result format that is nearest in value, subject to the current rounding mode, to what the result would be given unlimited range and precision”. For purposes of printf conversions, the result format is a decimal numeral. This definition effectively tells us there should be no arithmetic errors in the conversion.
In 5.2.4.2.2 9, the standard specifies some rounding methods, including “toward zero”, “to nearest”, “toward positive infinity”, and “toward negative infinity”, and it allows implementations to define other rounding modes. For the most part, the rounding mode governs the built-in arithmetic and cast operators. However, it should also govern printf conversions. (Ideally, it would also govern the math library routines such as sin, but these are rarely implemented to abide by the rounding mode.)
Thus, output of “.0039063” could result from using a rounding mode of “toward positive infinity”. However, I will assume your Microsoft implementation uses a default rounding mode of “to nearest”. You can test this by printing FLT_ROUNDS; it will be 1 for “to nearest”.
However, the C standard does not specify what the “to nearest” method does for ties, except that Annex F binds it to the IEEE-754 to-nearest method, which resolves ties in favor of the candidate with the even low digit. However, Annex F is optional; a C implementation is not required to use it.
Therefore, the behavior of rounding .00390625 to five significant digits with the “to nearest” method is technically not specified by the C standard.