Use pandas.shift() within a group

Question

I have a dataframe with panel data, let's say it's time series for 100 different objects:

object  period  value 
1       1       24
1       2       67
...
1       1000    56
2       1       59
2       2       46
...
2       1000    64
3       1       54
...
100     1       451
100     2       153
...
100     1000    21

I want to add a new column prev_value that will store previous value for each object:

object  period  value  prev_value
1       1       24     nan
1       2       67     24
...
1       99      445    1243
1       1000    56     445
2       1       59     nan
2       2       46     59
...
2       1000    64     784
3       1       54     nan
...
100     1       451    nan
100     2       153    451
...
100     1000    21     1121

Can I use .shift() and .groupby() somehow to do that?

Answer 1

Pandas' grouped objects have a groupby.DataFrameGroupBy.shift method, which will shift a specified column in each group n periods, just like the regular dataframe's shift method:

df['prev_value'] = df.groupby('object')['value'].shift()

For the following example dataframe:

print(df)

     object  period  value
0       1       1     24
1       1       2     67
2       1       4     89
3       2       4      5
4       2      23     23

The result would be:

     object  period  value  prev_value
0       1       1     24         NaN
1       1       2     67        24.0
2       1       4     89        67.0
3       2       4      5         NaN
4       2      23     23         5.0