Search code examples
pythonpandasdataframeconcatenationoverlap

Concatenate two dataframes, with some overlaping date index, with resulting dataframe taking "left" as default, except the "left" is NaN


Giving DF1 and DF2 below, how to get DFresult?

DF1:                        DF2:                        DFresult:       
Date | Value                Date | Value                Date | Value
------------                ------------                ------------
1-01-2019 | 1                                           1-01-2019 | 1       (no overlap, take the one that exists)
1-02-2019 | 1                                           1-02-2019 | 1       (no overlap, take the one that exists)
1-03-2019 | np.NaN          1-03-2019 | 2               1-03-2019 | 2       (left is NaN, take right)
1-04-2019 | 1               1-04-2019 | np.NaN          1-04-2019 | 1       (left is not NaN, take left)
1-05-2019 | np.NaN          1-05-2019 | np.NaN          1-05-2019 | np.NaN  (both NaN, keep it)
1-06-2019 | 1               1-06-2019 | 2               1-06-2019 | 1       (left is not NaN, take left)
                            1-07-2019 | 2               1-07-2019 | 2       (no overlap, take the one that exists)
                            1-08-2019 | 2               1-08-2019 | 2       (no overlap, take the one that exists)
                            1-09-2019 | 2               1-09-2019 | 2       (no overlap, take the one that exists)
                            1-10-2019 | 2               1-10-2019 | 2       (no overlap, take the one that exists)
                            1-11-2019 | 2               1-11-2019 | 2       (no overlap, take the one that exists)

What if I'd like to use a function to determine the overlaping decision? Example, take left if left is higher than right, or if left is NaN:

DF1:                        DF2:                        DFresult:       
Date | Value                Date | Value                Date | Value
------------                ------------                ------------
1-01-2019 | 1                                           1-01-2019 | 1       (no overlap, take the one that exists)
1-02-2019 | 1                                           1-02-2019 | 1       (no overlap, take the one that exists)
1-03-2019 | np.NaN          1-03-2019 | 2               1-03-2019 | 2       (left is NaN, take right)
1-04-2019 | 1               1-04-2019 | np.NaN          1-04-2019 | 1       (right is NaN, take left)
1-05-2019 | np.NaN          1-05-2019 | np.NaN          1-05-2019 | np.NaN  (both NaN, keep it)
1-06-2019 | 1               1-06-2019 | 2               1-06-2019 | 2       (left is not higher, take right)
1-06-2019 | 3               1-07-2019 | 2               1-07-2019 | 3       (left is higher, take left)
1-06-2019 | 1               1-08-2019 | 2               1-08-2019 | 2       (left is not higher, take right)
                            1-09-2019 | 2               1-09-2019 | 2       (no overlap, take the one that exists)
                            1-10-2019 | 2               1-10-2019 | 2       (no overlap, take the one that exists)
                            1-11-2019 | 2               1-11-2019 | 2       (no overlap, take the one that exists)

Solution

  • Try with

    out = pd.concat([DF1,DF2]).groupby('Date',as_index=False).max()
    # for your original one 
    
    #out = pd.concat([DF1,DF2]).groupby('Date',as_index=False).first()