I am trying to make an axis with tick marks labeled with numbers. Everything is working fine except for the last number generated for the axis is 0.099999 instead of 0.1. I have removed the line drawing parts from my code below, but left the part that generates the numbers. These are the numbers that I get from the below code: 1.0, 0.9, 0.8, 0.7, 0.6, 0.5, 0.4, 0.3, 0.2, 0.09999999 (should be 0.01). Is there anyway to round numbers or calculate the numbers a different way to avoid the rounding error?
/str 10 string def
/count 0 def
/totalUnits 1 def
/ticks 0.1 def
totalUnits ticks neg 0
{
/loopnum exch def
loopnum 10 str cvrs
/stringnumber exch def % string representing the number
/count count 0.1 add def
} for
As commented, you can use integers to avoid this quirk of floating-point numbers.
/str 10 string def
/count 0 def
/scaling .1 def
10 -1 1{
scaling mul
10 string cvs
%dup =
/stringnumber exch def
/count count 1 add def
} for
This same rounding issue affects all languages that use a binary floating-point representation for numbers since the fraction 1/10 has a repeatinging pattern in its binary representation. It's the same effect that gives you .333333... for 1/3 in decimal representation. Summing up these quantities loses bits from the bottom where the (input) values have been truncated.