Search code examples
pythonpandasfinance

How to check if time delta is greater than one minute in a dataframe?


I am trying to compare different time stamps in a dataframe and print an output when time difference is greater than one minute. This is the code I am trying to run:

for e in TestDF['date']:
    delta = TestDF.date.iloc[e+1] - TestDF.date.iloc[e]
    if delta > datetime.timedelta(minutes=1):
        print(TestDF.date.iloc[e+1])
        print(TestDF.date.iloc[e])

and this is the error I am getting:

ValueError: Cannot add integral value to Timestamp without freq.

However this seems to work:

TimeDifference = TestDF.date.iloc[4]-TestDF.date.iloc[3]
if TimeDifference == datetime.timedelta(minutes=1):
    print(TimeDifference)

output: 0 days 00:01:00

Any help will be appreciated.

Thank you,

Here is some sample data:

        date             open    high    low     close
0   2020-01-28 07:00:00 311.83  311.89  311.62  311.81
1   2020-01-28 07:01:00 311.80  311.98  311.80  311.85
2   2020-01-28 07:02:00 311.90  312.00  311.88  311.98
3   2020-01-28 07:03:00 312.00  312.02  311.99  311.99
4   2020-01-28 07:04:00 312.00  312.00  311.91  311.91

Solution

  • You just need to use a combination of shift and boolean filtering:

    note i've changed your last row to show a difference bigger than 1 minute.

    print(df)
                     date    open    high     low   close
    0 2020-01-28 07:00:00  311.83  311.89  311.62  311.81
    1 2020-01-28 07:01:00  311.80  311.98  311.80  311.85
    2 2020-01-28 07:02:00  311.90  312.00  311.88  311.98
    3 2020-01-28 07:03:00  312.00  312.02  311.99  311.99
    4 2020-01-28 07:06:00  312.00  312.00  311.91  311.91
    

    first we ensure our date is a proper datetime

    df['date'] = pd.to_datetime(df['date'])
    
    
    ((df['date'] - df['date'].shift()) / np.timedelta64(1,'m')) > 1)
    out:
    0    False
    1    False
    2    False
    3    False
    4     True
    Name: date, dtype: bool
    

    you can then filter it by your data

    print(df[((df['date'] - df['date'].shift()) / np.timedelta64(1,'m')) > 1])
      date                 open   high     low   close
    4 2020-01-28 07:06:00  312.0  312.0  311.91  311.91