Search code examples
pythonpandasdataframemergeconcatenation

Merging dataframes with hierarchcial index?


Question

  • How to merge two dataframes, using hierarchical index?
  • The index of my dataframes are datetime.datetime and datetime.time

Code example

import pandas as pd
import numpy as np
import datetime

BTC_df = pd.DataFrame(np.array([[datetime.time(19,15,0),1.05,'BTC'],[datetime.time(9,20,0),1.2,'BTC']]),
                      columns = ['time','return','coin'],
                      index = [[datetime.date(2020,5,1), datetime.date(2020,5,2)]])

ETH_df = pd.DataFrame(np.array([[datetime.time(8,30,0),1.1,'ETH'],[datetime.time(17,30,0),0.9,'ETH']]),
                      columns = ['time','return','coin'],
                      index = [[datetime.date(2020,5,1), datetime.date(2020,5,2)]])

BTC_df 
>>>               time    return   coin
   2020-05-01   19:15:00    1.05    BTC
   2020-05-02   09:20:00    1.2     BTC

ETH_df 
>>>                time    return   coin
   2020-05-01   08:30:00    1.1     ETH
   2020-05-02   17:30:00    0.9     ETH

My desired output is as follows, using hierarchical index (MultiIndex)

merged_df
>>>               time    return   coin
   2020-05-01   08:30:00    1.1     ETH
                19:15:00    1.05    BTC
   2020-05-02   09:20:00    1.2     BTC
                17:30:00    0.9     ETH

Solution

  • You can use set_index with append to make time :

    pd.concat([BTC_df,ETH_df]).set_index('time',append=True).sort_index()
    

    Output:

                        return coin
               time                
    2020-05-01 08:30:00    1.1  ETH
               19:15:00   1.05  BTC
    2020-05-02 09:20:00    1.2  BTC
               17:30:00    0.9  ETH