I have a pandas dataframe, and I want to filter out some records that don't satisfy certain criteria:
cond1=df['ap_lo'] <= df['ap_hi']
cond2=df['height'] >= df['height'].quantile(q=0.025)
cond3=df['height'] <= df['height'].quantile(q=.957)
cond4=df['weight'] >= df['weight'].quantile(q=0.025)
cond5=df['weight'] <= df['weight'].quantile(q=0.975)
This gives the following percentage of the original data:
filtered_df=df[cond1 & cond2 & cond3 & cond4 & cond5]
print(filtered_df.shape[0] / df.shape[0])
>>>0.8865
What surprised me is that when I try the same but using parenthesis between conditions instead of predefining the conditions variables I get different results:
filtered_df = df[(df['ap_lo'] <= df['ap_hi']) &
(df['height'] >= df['height'].quantile(0.025)) &
(df['height'] <= df['height'].quantile(0.975)) &
(df['weight'] >= df['weight'].quantile(0.025)) &
(df['weight'] <= df['weight'].quantile(0.975))]
print(filtered_df.shape[0] / df.shape[0])
>>> 0.9037
What is going on? I don't think that is something related to the order in which these operations are performed because AND is associative and commutative as far as I remember...
@luuk's comment is correct; the problem is you've transposed digits in your second definition of the third condition.
import pandas as pd
import numpy as np
df = pd.DataFrame(data=np.random.uniform(size=(250,4)), columns=['ap_hi', 'ap_lo', 'height', 'weight'])
a = (df['height'] <= df['height'].quantile(q=0.975))
b = (df['height'] <= df['height'].quantile(q=0.957))
print((a == b).all())