Search code examples
floating-pointsap-ase

sybase 15.7 float to numeric rounding error


Can anyone explain the following, which to me simply looks like a bug:

select convert(numeric(8,4), convert(float, '12.4155499999999996418864611769'))

12.4156

Surely the answer should be 12.4155. To double check I created a table, MyTab, with a single sybase "float" column and inserted "12.4155499999999996418864611769". Then using convert dumped the hex and got 0x4028D4C2F837B4A2. This is indeed the IEEE 754 double precision representation of the string I inserted. Then if I (as above) convert that to a numeric(8,4) it gives 12.4156.

Is this wrong as I think it is, or am I going mad?


Solution

  • You are likely facing a double-rounding bug in the conversion from double-precision to decimal in Sybase.

    It is probable that the algorithm used is to first print the IEEE 754 number (in your case, represented by 0x4028D4C2F837B4A2) with, say, 17 decimal digits of precision, on the basis that this number of digits is in some sense the “decimal precision” of IEEE 754's double-precision format. For your example, the result of this conversion to decimal is 12.415550000000000.

    Then, since you asked for 4 decimal digits after the dot, the intermediate decimal representation is rounded to that, under either ties-go-up or ties-go-to-nearest-even. Both are wrong anyway because the first rounding has lost some crucial information (namely, that the original number was slightly below 12.41555, not exactly it).

    I am not familiar with Sybase, but in all likelihood there is no easy workaround. You could require a conversion to decimal with enough extraneous digits to avoid the problem (that is, 21 extra digits, see below) but you can expect the digits after the 17th to print as “0” for the same reason that you are getting double-rounded decimal conversions in the first place. You could subtract 12 from the double-precision number you have (that operation happens to be exact, that is, not to introduce any approximation) in order to gain a few digits in the conversion to decimal, but that will not be enough to ensure the correct rounding of all double-precision values.

    Notes:

    1. For reference, the exact value of the double-precision number we are talking about is 12.415549999999999641886461176909506320953369140625.

    2. Converting, for the first conversion, with more than 20 decimal digits beyond what was required for the second rounding would have made the result safe from any noticeable effects of the “double rounding”, according to Mark Dickinson's answer to this question. The number in your question is a good candidate for the maximum number of consecutive nines in a double, but it pales in front of Mark's 2.12818792307269553358078502102171540639252016258831784842556110831434197718043638405555406495645619729155240037555858106390933161420388023706431461384056688295540725831155392678607931808851292893574214797681879999999999999999999941026584542575391157788777223962620780080784703190447744595561259568772261019375946489162743091583251953125E-122.