Search code examples
pythonscipyoutliersanomaly-detectionnoise-generator

How can generate impulse as outliers on periodic or sequenced-based data for doing experiments via outliers detection methods?

I want to carry out some experiments on some time-series data with the KM approach has been suggested in this paper. The problem is I don't access the data in the paper but I have similar type data which has no outliers and I need to generate some outliers artificially/manually in form of the impulse (top & bottom) so that meet the requirements as it is shown in the following picture from this paper achievement:

img

In the worst case, I was wondering if I could generate something like this on the periodic perfect train of pulse or Sin function to apply to the available data. So far, my implementation is limited to these post1, post2 and Welch but in order to generate right outliers, I was thinking of identifying/detecting high and low states and then add outliers frequently for further experiments of outlier detectors.

I'm unsure that convolving impulses as noise to target outliers is the right way to contaminate periodic data I access or not.

So far I have generated a train of impulses but I have no idea how I can include to one periodic signal or data:

import scipy.signal as signal
import matplotlib.pyplot as plt
imp = signal.unit_impulse(200, [10,50,60])

fig, ax = plt.subplots(1,1, figsize=(8,6))

ax.set_xlabel('Cycles')
plt.plot(imp)
plt.title('Generating impulse outliers in desired cycles: 10, 50 & 60')
plt.ylim((0.0, 1.4))
plt.show()

img

Any helps will be appreciated since many people are working on topics like Anomaly and outlier detection.

Solution

  • You can generate a signal with eg.: numpy Python module. And you can add your impulses to this signal (of course, if the dimensions are correct). I have written an example for your where I have generated a Sinus signal with numpy and I have added the impulses with the signal.unit_impulse() like in your question. I have added several comments to code for the better understanding.

    Code:

    import numpy as np
import matplotlib.pyplot as plt
import scipy.signal as signal

positive_impulses = signal.unit_impulse(200, [10, 50, 60])  # Generate positive impulses
negative_impulses = signal.unit_impulse(200, [80, 100, 150])*-1  # Generate negative impulses

# Generate the Sinus signal.
t = np.linspace(1, 200, 200)
x_sin_sig = np.sin(t / (2 * np.pi))
plt.subplot(4, 1, 1)
plt.plot(t, x_sin_sig)
plt.title("Signal")
plt.ylabel("Sin")


plt.subplot(4, 1, 2)
plt.plot(t, x_sin_sig + positive_impulses)  # Add the positive impulses to the original signal
plt.title("Signal with positive impulses")

plt.subplot(4, 1, 3)
plt.plot(t, x_sin_sig + negative_impulses)  # Add the negative impulses to the original signal
plt.title("Signal with negative impulses")

plt.subplot(4, 1, 4)
plt.plot(t, x_sin_sig + positive_impulses + negative_impulses)  # Add the both impulses to the original signal
plt.title("Signal with different impulses")

plt.tight_layout()
plt.show()

    Output:

    Output image

    Note:

    Probably more efficient to generate random impulses for your signal. You can do it with the random module.

    import random

positive_impulses = signal.unit_impulse(200, random.sample(range(200), 5))  # Generate positive impulses
negative_impulses = signal.unit_impulse(200, random.sample(range(200), 5))*-1  # Generate negative impulses

    The random.sample(range(200), 5) will return a list of 5 numbers selected from the range 0 to 200, without duplicates.

    An example with random impulses:

    Random impulses