Search code examples
pythonpython-3.xpandasdatetimeepoch

How to extract only the epoch details and leave other things out in pandas dataframe?


I have a dataset with time in the epoch. I need to extract the epoch time and convert it into normal DD MM YYYY format along with time details in HH: MM format. The column is like this:-

Index       Date                                                                  
0        {'$date': {'$numberLong': '1562005805010'}}   

I have tried using regex, extract and replace methods but they convert the date column to NaN

df1['date'] = df1['date'].str.extract('(\d+)', expand=False)

I want only epochs to be displayed so that they can be converted to date and time. Here is the column that I have


Solution

  • If values are strings first convert it to dictioanries by ast.literal_eval and then select:

    print (type(df['Date'].iat[0]))
    <class 'str'>
    
    import ast
    
    s = df['Date'].apply(lambda x: ast.literal_eval(x)['$date']['$numberLong'])
    

    If values are nested dicts only select by keys:

    print (type(df['Date'].iat[0]))
    <class 'dict'>
    
    s = df['Date'].apply(lambda x: x['$date']['$numberLong'])
    

    And last use to_datetime with unit parameter:

    print (s)
    0    1562005805010
    Name: Date, dtype: object
    
    df['Date'] = pd.to_datetime(s, unit='ms')
    print (df)
       Index                    Date
    0      0 2019-07-01 18:30:05.010