I am trying to compare different time stamps in a dataframe and print an output when time difference is greater than one minute. This is the code I am trying to run:
for e in TestDF['date']:
delta = TestDF.date.iloc[e+1] - TestDF.date.iloc[e]
if delta > datetime.timedelta(minutes=1):
print(TestDF.date.iloc[e+1])
print(TestDF.date.iloc[e])
and this is the error I am getting:
ValueError: Cannot add integral value to Timestamp without freq.
However this seems to work:
TimeDifference = TestDF.date.iloc[4]-TestDF.date.iloc[3]
if TimeDifference == datetime.timedelta(minutes=1):
print(TimeDifference)
output: 0 days 00:01:00
Any help will be appreciated.
Thank you,
Here is some sample data:
date open high low close
0 2020-01-28 07:00:00 311.83 311.89 311.62 311.81
1 2020-01-28 07:01:00 311.80 311.98 311.80 311.85
2 2020-01-28 07:02:00 311.90 312.00 311.88 311.98
3 2020-01-28 07:03:00 312.00 312.02 311.99 311.99
4 2020-01-28 07:04:00 312.00 312.00 311.91 311.91
You just need to use a combination of shift and boolean filtering:
note i've changed your last row to show a difference bigger than 1 minute.
print(df)
date open high low close
0 2020-01-28 07:00:00 311.83 311.89 311.62 311.81
1 2020-01-28 07:01:00 311.80 311.98 311.80 311.85
2 2020-01-28 07:02:00 311.90 312.00 311.88 311.98
3 2020-01-28 07:03:00 312.00 312.02 311.99 311.99
4 2020-01-28 07:06:00 312.00 312.00 311.91 311.91
first we ensure our date is a proper datetime
df['date'] = pd.to_datetime(df['date'])
((df['date'] - df['date'].shift()) / np.timedelta64(1,'m')) > 1)
out:
0 False
1 False
2 False
3 False
4 True
Name: date, dtype: bool
you can then filter it by your data
print(df[((df['date'] - df['date'].shift()) / np.timedelta64(1,'m')) > 1])
date open high low close
4 2020-01-28 07:06:00 312.0 312.0 311.91 311.91