Search code examples
pandasdataframecomparevalueerror

Pandas ValueError: Can only compare identically-labeled Series objects

df = pd.DataFrame([{'Instrument':'BHP', 'Date':'2012-04-18', 'Time':'09:59:34.160', 'Milliseconds':35974160 , 'RecordType':'ENTER', 'Value':36597.95},
                    {'Instrument':'BHP', 'Date':'2012-04-18', 'Time':'09:59:34.566', 'Milliseconds':35974566 , 'RecordType':'DELETE', 'Value':175.70},
                    {'Instrument':'BHP', 'Date':'2012-04-18', 'Time':'09:59:37.832', 'Milliseconds':35977832 , 'RecordType':'DELETE', 'Value':1093470.00},
                    {'Instrument':'BHP', 'Date':'2012-04-18', 'Time':'09:59:37.841', 'Milliseconds':35977841 , 'RecordType':'DELETE', 'Value':25799.34},
                    {'Instrument':'BHP', 'Date':'2012-04-18', 'Time':'09:59:38.846', 'Milliseconds':35978846 , 'RecordType':'ENTER', 'Value':2460.15},
                    {'Instrument':'BHP', 'Date':'2012-04-18', 'Time':'09:59:45.015', 'Milliseconds':35985015 , 'RecordType':'DELETE', 'Value':6731.00},
                    {'Instrument':'BHP', 'Date':'2012-04-18', 'Time':'09:59:47.024', 'Milliseconds':35987024 , 'RecordType':'OPEN', 'Value':np.nan}])```

I have the above DataFrame. My aim is to obtain the sum of values with RecordType DELETE from 10 seconds before the OPEN to OPEN. I tried the following codes:

opening_time = df[df.RecordType=='OPEN']
ten_seconds_before_open = opening_time['Milliseconds'] - 10*1000
delete_type = df[df.RecordType=='DELETE']
sum_delete = delete_type[delete_type.Milliseconds >= ten_seconds_before_open].Value.sum()
print(sum_delete)

However, it returns ValueError: Can only compare identically-labeled Series objects. May I know what is the best solution for this?

In fact, I actually have millions of rows containing many Instrument and Date. I was trying to code to obtain the sum of DELETE values for each Instrument for each Date,

def sum_delete_type(df):
     opening_time = df[df.RecordType=='OPEN']
     ten_seconds_before_open = opening_time['Milliseconds'] - 10*1000
     delete_type = df[df.RecordType=='DELETE']
     sum_delete = delete_type[delete_type.Milliseconds >= ten_seconds_before_open].Value.sum()
     return sum_delete

df.groupby(['Instrument', 'Date']).apply(sum_delete_type)

but it didn't work. Please help. Thank you.

Solution

  • How about this one

    opening_time = df[df.RecordType=='OPEN']
ten_seconds_before_open = opening_time['Milliseconds'] - 10*1000
delete_type = df[df.RecordType=='DELETE']
y=[]
for x in ten_seconds_before_open:
    y.extend(delete_type[delete_type.Milliseconds >= x].index.tolist())
y=list(set(y))
delete_type[delete_type.index.isin(y)]['Milliseconds'].sum()