Search code examples
pythonpandaslinear-interpolationpandas-resample

Pandas linear interpolation for geometrical X-Y data seems to ignore points

I am trying to upsample my dataframe in pandas (from 50 Hz to 2500 Hz). I have to upsample to match a sensor that was sampled at this higher frequency. I have points in x, y, z coming from a milling machine. When I am plotting the original data the lines look straight, as I would expect.

Original data Original data zoomed in

I am interpolating the dataframe like this:

df.drop_duplicates(subset='time', inplace=True)
df.set_index('time', inplace=True)
df.index = pd.DatetimeIndex(df.index)
upsampled = new_df.resample('0.4ms').interpolate(method='linear')
plt.scatter(upsampled['X[mm]'], upsampled['Y[mm]'], s=0.5)
plt.plot()

I also tried with

upsampled = df.resample('0.4L').interpolate(method='linear')

I expect the new points to always come between the original points. Since I am going from 50 Hz to 2500 Hz, I expect 50 points uniformly spaced between each pair of points in the original data. However, it seems that some of the original points are ignored, as can be seen in the picture below (the second picture is zoomed in on a particularly troublesome spot).

Upsample data superimposed Upsample data superimposed and zoomed

This figure shows the original points in orange and the upsampled, interpolated points in blue (both are scattered, although the upsampled points are so dense it appears as a plot). The code for this is shown below.

upsampled = df.resample('0.4ms').interpolate(method='linear')
plt.scatter(upsampled['X[mm]'], upsampled['Y[mm]'], s=0.5, c='blue') 
plt.scatter(df['X[mm]'], df['Y[mm]'], s=0.5, c='orange')
plt.gca().set_aspect('equal', adjustable='box')
fig.show()

Any ideas how I could make the interpolation work?

Solution

  • Most likely the problem is that the timestamps in the original and resampled DataFrames are not aligned, so when resampling we need to specify how to deal with that.

    Since the original is at 50 Hz and the resampled is at 2500 Hz, simply taking mean should fix it:

    upsampled = new_df.resample('0.4ms').mean().interpolate(method='linear')

    Unfortunately, without having any sample data, I cannot verify that it works. Please let me know if it does help