Search code examples
python-3.xpandastime-seriespython-datetime

Pandas DateTime get duration of file


My data files consist of roughly 1 million rows of time-series data. It has been read into Python using df = pd.read_csv(...). I am looking for a way to get the duration of the file (in seconds), the output I am looking for is just one number to give the duration

Below shows the first and last 5 entries to show the structure of the data:

df.head(5)

                                 X         Y         Z
 TimeStamp                                            
 2017-05-12 11:03:39.560  0.185310 -0.168226  0.385064
 2017-05-12 11:03:39.570  0.184273 -0.290579  0.497026
 2017-05-12 11:03:39.580  0.188649 -0.456002  0.601236
 2017-05-12 11:03:39.590  0.195188 -0.629775  0.679267
 2017-05-12 11:03:39.600  0.196400 -0.789999  0.729308

df.tail(5)

                                 X         Y         Z
 TimeStamp                                            
 2017-05-12 13:18:59.950 -0.045288 -0.018508  1.010065
 2017-05-12 13:18:59.960 -0.045412 -0.018438  1.009695
 2017-05-12 13:18:59.970 -0.045671 -0.018282  1.009768
 2017-05-12 13:18:59.980 -0.045889 -0.018029  1.010952
 2017-05-12 13:18:59.990 -0.045657 -0.017709  1.013374

Solution

  • IIUC, let's try, given TimeStamp is a DatetimeIndex: First let's get you index into datetime:

    df.index = pd.to_datetime(df.index)
    
    
    df.reset_index()['TimeStamp'].diff().sum().total_seconds()
    

    OR

    (df.index[-1] - df.index[0]).total_seconds()