Search code examples
pythonmathscipydata-science

How to detect horizontal line (stagnation) in data?


How can I detect horizontal lines in my data in python?

I have used function scipy.signal.find_peaks() to find local minima and maxima. I can use that to separate ascending and descending parts. But I need to isolate the peaks from the flat line (marked with red circes in image).

What method should I use? Is there any library that could do that?

Graph with red markers around flat horizontal lines


Solution

  • I would write some algorithm to find the differences between points in the graph.

    pandas.DataFrame.diff is useful

    difference = df.diff(periods=-1)
    

    Above line gets you the difference between each row and the next

    Use some threshold number, lets say 0.1.

    threshold = 0.1
    

    Check if the difference between points is > threshold

    If difference < threshold, then continue on the next point until diff > threshold. (this means you're at the end of that flat line)

    Continue looping through all your data until all flat lines have been discovered. Then do whatever operations with that data.

    Something like this: (probably won't work, basically pseudocode)

    flat_lines = []
    for point in difference:
        if difference <= threshold:
            #Change in point a to point b is less than 0.1
            #start = index of start of flat line
            flat_lines.append(start)
            continue
        elif difference > threshold:
            #End of flatline
            #store whatever data you need
            #end= index of endof flat line
            flat_lines.append(end)
            continue
    

    These posts may also help: A simple algorithm to detect flat segments in noisy signals

    Replace "flatline" repeated data in Pandas series with nan