Search code examples
pythonpandasdataframebit-manipulationoperator-keyword

Pandas bitwise filtering gives different results with and without parenthesis


I have a pandas dataframe, and I want to filter out some records that don't satisfy certain criteria:

cond1=df['ap_lo'] <= df['ap_hi']
cond2=df['height'] >= df['height'].quantile(q=0.025)
cond3=df['height'] <= df['height'].quantile(q=.957)
cond4=df['weight'] >= df['weight'].quantile(q=0.025)
cond5=df['weight'] <= df['weight'].quantile(q=0.975)

This gives the following percentage of the original data:

filtered_df=df[cond1 & cond2 & cond3 & cond4 & cond5]
print(filtered_df.shape[0] / df.shape[0]) 
>>>0.8865

What surprised me is that when I try the same but using parenthesis between conditions instead of predefining the conditions variables I get different results:

filtered_df = df[(df['ap_lo'] <= df['ap_hi']) & 
                 (df['height'] >= df['height'].quantile(0.025)) &
                 (df['height'] <= df['height'].quantile(0.975)) &
                 (df['weight'] >= df['weight'].quantile(0.025)) & 
                 (df['weight'] <= df['weight'].quantile(0.975))]
print(filtered_df.shape[0] / df.shape[0])
>>> 0.9037

What is going on? I don't think that is something related to the order in which these operations are performed because AND is associative and commutative as far as I remember...


Solution

  • @luuk's comment is correct; the problem is you've transposed digits in your second definition of the third condition.

    import pandas as pd
    import numpy as np
    
    df = pd.DataFrame(data=np.random.uniform(size=(250,4)), columns=['ap_hi', 'ap_lo', 'height', 'weight'])
    
    a = (df['height'] <= df['height'].quantile(q=0.975))
    b = (df['height'] <= df['height'].quantile(q=0.957))
    print((a == b).all())