Detrend or Filter a sawtooth signal (Python)

Question

I was trying to remove the trend (sawtooth) of a signal by using a filfilt filter without good results. Here the data

core_3 = pd.read_csv(filename_3, sep=";",header=0,index_col = 0,
                     parse_dates=True, names='date','temp'], infer_datetime_format=True, dayfirst=True)

core_7 = pd.read_csv(filename_7 ,sep=";",header=0,index_col = 0,
                    parse_dates=True, names=['date','temp'], infer_datetime_format=True,dayfirst=True)

When I applied a Butterworth filter

b3, a3 = sg.butter(1, 0.045)
y3     = sg.filtfilt(b3, a3, core_3.temp)
b7, a7 = sg.butter(1, 0.030)
y7     = sg.filtfilt(b7, a7, core_7.temp)

The result is

As you can see for 3 THz signal there is a sawtoohth-trend. At 21:45 the signal have a perturbation (I wanna study that perturbation). That perturbation is clearly observed at 7 THz. At 3 THz is was observed a interruption of the sawtooth. So, I need to detrend or filter this signal. I tried using a filtfilt filter, but I dont know if is more convenient to use scipy.detrend.

Answer 1

Neither filtering nor simple detrending will do you any good with this signal. The first problem is that the trend is somewhat periodic. The second problem is that the periodicity is not stationary. I believe that linear methods will not solve the problem.

I suggest you do the following:

1. remove the jumps ("unroll the sawtooth")
2. then detrend the signal, with a low-pass filter or whatever

Here is an example:

import numpy as np
import scipy.signal as sps
import matplotlib.pyplot as plt

np.random.seed(123)

# create an example signal
x = []
ofs = 3.4
slope = 0.002
for t in np.linspace(0, 100, 1000):
    ofs += slope    
    x.append(np.sin(t*2) * 0.1 + ofs)
    if x[-1] > 4:
        ofs =3.2
        slope = np.random.rand() * 0.003 + 0.002
x = np.asarray(x)    
plt.plot(x, label='original')

# detect and remove jumps
jmps = np.where(np.diff(x) < -0.5)[0]  # find large, rapid drops in amplitdue
for j in jmps:
    x[j+1:] += x[j] - x[j+1]    
plt.plot(x, label='unrolled')

# detrend with a low-pass
order = 200
x -= sps.filtfilt([1] * order, [order], x)  # this is a very simple moving average filter
plt.plot(x, label='detrended')

plt.legend(loc='best')
plt.show()