Search code examples
matlabsignalssignal-processingnoisenoise-reduction

How to identify decreasing intervals in noisy data?

I have multiple segments of data that look similar to the one in the image:

Decay

In the signal there is some stability, then some period where the signal is decreasing, and then the signal stabilizes again. I am trying to find a way to identify the intervals where the data is generally decreasing (the area between the red lines in the image), but the noise in the data has made this difficult for me.

I have tried using variance and a comparison of a simple moving average and exponential weighted moving average to find the points at which the decrease begins and ends, but the noise is messing things up, so my main questions are for any recommendations for a good way to smooth out the noise or even better if there are any suggestions on how to identify those decreasing intervals.

Solution

  • Broadly, you have three stages

    1. Signal is stable / horizontal
    2. Signal is changing
    3. Signal is stable / horizontal

    We need to characterise these mathematically, we can do that many ways, but one which is somewhat intuitive is using the rolling standard deviation

    1. Stable signal ⟹ small standard deviation over neighbouring n samples
    2. Changing signal ⟹ large standard deviation over neighbouring n samples
    3. Stable signal ⟹ small standard deviation over neighbouring n samples

    We can easily compute the rolling standard deviation with a loop.

    First, some dummy data:

    rng(0);                          % fix random seed for repeatable example
N = 1e3;                         % number of samples
x = linspace(0,1,N);             % fixed rate x axis data
y = rand(1,N)*0.4 + (x>0.5)*0.5; % noisy data with a step change in middle
y = movmean( y, N*0.08 );        % smooth out the data a bit

    Now compute the std. dev. over some window. The length of this window will need tuning for your data, depending on how abrupt your step changes are relative to the sample rate.

    n = 50; % window size
sd = zeros(size(y));
for ii = (n/2):(N-n/2)
    % std dev over 'n' samples, centred around the iith point
    sd(ii) = std(y(ii-(n/2)+1:ii+(n/2)));  
end

    Using a simple threshold which will also need tuning to your data, we can identify the areas of high-change:

    thrDetect = 0.05;                % threshold to detect change from std. dev.
bDetect = sd > thrDetect;        % Boolean detection array

    Plotting, we can check this works:

    plot

    figure(1); clf; subplot( 2, 1, 1 ); hold on;
plot( x, y, '.', 'displayname', 'data' );
plot( x, bDetect, 'k', 'displayname', 'std. dev. over threshold' );
legend('show', 'location', 'best'); grid on;
subplot( 2, 1, 2 ); hold on;
plot( x, sd, 'displayname', 'Rolling std. dev' );
yline( thrDetect, 'displayname', 'threshold' );
legend('show', 'location', 'best'); grid on;