Search code examples
pythonpython-3.xnumpyfloating-pointprecision

Why do a//b and np.floor(a/b) produce different results?


I stumbled upon a weird edge behaviour of python 3's "flooring division" using //. After some calculations I arrive at the formula (a1 - a2) // b. Using

a1 = 226.72560000000001
a2 = 0.2856000000000165
b = 1.02
# all types are np.float64

I get a different result than had I used np.floor():

>>> (a1 - a2) // b
221.0
>>> np.floor((a1 - a2) / b)
222.0

This difference obviously stems from floating point calculation, but why do they behave differently? Using a higher precision calculator such as ke!san I can verify that //'s behaviour is mathematically correct, while for my calculation np.floor() would deliver the correct result.


Solution

  • If x is very close to an exact integer multiple of y, it’s possible for x//y to be one larger than (x-x%y)//y due to rounding. In such cases, Python returns the latter result, in order to preserve that divmod(x,y)[0] * y + x % y be very close to x

    link to doc:

    https://docs.python.org/3/reference/expressions.html#id10

    What should the output of:

    3.3//1.1
    
    3.0 right?
    
    wrong it is 2.0
    

    So, when the numerator is very close to the multiple of denominator

    x//y will return (x-x%y)//y:

    a1 = 226.72560000000001
    a2 = 0.2856000000000165
    b = 1.02
    
    x = a1-a2
    y = 1.02
    
    (x-x%y)//y
    
    #output
    221.0