Search code examples
pandassample

Resample and fill df - pandas


I'm hoping to resample a df and ffill the unspecified rows. Below is what I'm currently doing

d1 = ({   
    'Time' : ['2019-08-02 09:50:11.5','2019-08-02 09:50:11.6','2019-08-02 09:50:11.8','2019-08-02 09:50:12.0','2019-08-02 09:50:11.5','2019-08-02 09:50:11.6','2019-08-02 09:50:11.8','2019-08-02 09:50:12.0'],
    'Value' : ['A','A','A','A','B','B','B','B'],
   })

df1 = pd.DataFrame(data=d1)

df1['Time'] = pd.to_datetime(df1['Time'])

df1['Time'] = df1.set_index('Time').resample('0.1S').ffill().reset_index()

Out:

ValueError: cannot reindex from a duplicate axis

Intended:

                    Time Value
0  2019-08-02 09:50:11.5     A
1  2019-08-02 09:50:11.6     A
2  2019-08-02 09:50:11.7     A
3  2019-08-02 09:50:11.8     A
4  2019-08-02 09:50:11.9     A
5  2019-08-02 09:50:12.0     A                        
6  2019-08-02 09:50:11.5     B
7  2019-08-02 09:50:11.6     B
8  2019-08-02 09:50:11.7     B
9  2019-08-02 09:50:11.8     B
10  2019-08-02 09:50:11.9     B
11  2019-08-02 09:50:12.0     B

Solution

  • Use groupby and asfreq() to upsample

    d1 = ({   
        'Time' : ['2019-08-02 09:50:11.5','2019-08-02 09:50:11.6','2019-08-02 09:50:11.8','2019-08-02 09:50:12.0','2019-08-02 09:50:11.5','2019-08-02 09:50:11.6','2019-08-02 09:50:11.8','2019-08-02 09:50:12.0'],
        'Value' : ['A','A','A','A','B','B','B','B'],
       })
    
    df1 = pd.DataFrame(data=d1)
    
    df1['Time'] = pd.to_datetime(df1['Time'])
    
    df1 = df1.set_index('Time')
    
    df1 = df1.groupby('Value', group_keys=False).resample('0.1S').asfreq().ffill().reset_index()
    
    df1
    
    Out[1]: 
                          Time Value
    0  2019-08-02 09:50:11.500     A
    1  2019-08-02 09:50:11.600     A
    2  2019-08-02 09:50:11.700     A
    3  2019-08-02 09:50:11.800     A
    4  2019-08-02 09:50:11.900     A
    5  2019-08-02 09:50:12.000     A
    6  2019-08-02 09:50:11.500     B
    7  2019-08-02 09:50:11.600     B
    8  2019-08-02 09:50:11.700     B
    9  2019-08-02 09:50:11.800     B
    10 2019-08-02 09:50:11.900     B
    11 2019-08-02 09:50:12.000     B