sharp step detection in one-dimensional data

Question

My Question is basically divided into two parts. I have a one-dimensional data set, that I want do smooth on the one hand to get rid of noise and on the other hand I want to detect (large) steps very precisly within the data.

1) Regarding the step detection part I already found an interesting entry here Step detection in one-dimensional data but this method seems not to work properly for my data.

So my original data looks like: enter image description here

When I apply the above mentioned method the result is the following: enter image description here

2) Regarding the data-smoothing part I already tried some data smoothing like applying the gaussian_filter1d from scipy. But of course the smoother my data get the less precise the step will be detected (or am I wrong in this part?). Is there a better way to smooth the data but keep the steps sharp?

the original data would be:

[441.0, 414.0, 389.0, 430.0, 403.0, 402.0, 475.0, 417.0, 535.0, 476.0, 461.0, 415.0, 476.0, 442.0, 411.0, 422.0, 417.0, 451.0, 428.0, 455.0, 446.0, 414.0, 411.0, 413.0, 412.0, 424.0, 454.0, 439.0, 422.0, 417.0, 449.0, 417.0, 445.0, 411.0, 388.0, 420.0, 432.0, 457.0, 421.0, 415.0, 486.0, 449.0, 419.0, 453.0, 424.0, 451.0, 432.0, 464.0, 430.0, 469.0, 404.0, 461.0, 443.0, 440.0, 407.0, 452.0, 424.0, 426.0, 451.0, 423.0, 479.0, 407.0, 419.0, 435.0, 463.0, 455.0, 420.0, 413.0, 441.0, 451.0, 398.0, 433.0, 449.0, 451.0, 441.0, 450.0, 412.0, 440.0, 419.0, 408.0, 393.0, 395.0, 409.0, 450.0, 416.0, 398.0, 443.0, 403.0, 420.0, 411.0, 397.0, 416.0, 455.0, 392.0, 441.0, 422.0, 398.0, 445.0, 491.0, 405.0, 410.0, 381.0, 427.0, 403.0, 408.0, 427.0, 441.0, 464.0, 429.0, 420.0, 463.0, 405.0, 422.0, 408.0, 421.0, 464.0, 442.0, 429.0, 477.0, 407.0, 438.0, 453.0, 359.0, 435.0, 452.0, 443.0, 371.0, 503.0, 464.0, 466.0, 446.0, 465.0, 409.0, 440.0, 415.0, 384.0, 462.0, 446.0, 454.0, 440.0, 490.0, 400.0, 430.0, 416.0, 466.0, 436.0, 451.0, 408.0, 423.0, 449.0, 396.0, 417.0, 409.0, 446.0, 408.0, 426.0, 401.0, 417.0, 410.0, 361.0, 381.0, 416.0, 443.0, 413.0, 449.0, 411.0, 425.0, 439.0, 392.0, 464.0, 432.0, 387.0, 408.0, 454.0, 455.0, 449.0, 420.0, 379.0, 446.0, 400.0, 436.0, 461.0, 451.0, 433.0, 417.0, 415.0, 449.0, 397.0, 390.0, 381.0, 416.0, 411.0, 407.0, 420.0, 474.0, 444.0, 390.0, 413.0, 410.0, 428.0, 425.0, 405.0, 398.0, 406.0, 431.0, 420.0, 468.0, 463.0, 431.0, 415.0, 381.0, 427.0, 438.0, 431.0, 388.0, 440.0, 467.0, 453.0, 441.0, 373.0, 418.0, 450.0, 395.0, 407.0, 433.0, 425.0, 462.0, 392.0, 424.0, 435.0, 429.0, 409.0, 430.0, 410.0, 394.0, 407.0, 412.0, 420.0, 432.0, 419.0, 415.0, 338.0, 397.0, 436.0, 422.0, 409.0, 431.0, 422.0, 437.0, 377.0, 493.0, 417.0, 416.0, 468.0, 468.0, 428.0, 459.0, 410.0, 462.0, 391.0, 433.0, 410.0, 388.0, 397.0, 421.0, 416.0, 429.0, 410.0, 447.0, 368.0, 397.0, 405.0, 421.0, 393.0, 446.0, 404.0, 414.0, 418.0, 426.0, 408.0, 389.0, 426.0, 442.0, 476.0, 409.0, 426.0, 392.0, 402.0, 435.0, 415.0, 392.0, 435.0, 442.0, 461.0, 422.0, 389.0, 445.0, 423.0, 379.0, 435.0, 416.0, 454.0, 490.0, 446.0, 457.0, 432.0, 413.0, 407.0, 465.0, 451.0, 427.0, 459.0, 446.0, 419.0, 413.0, 418.0, 409.0, 413.0, 442.0, 454.0, 405.0, 397.0, 445.0, 441.0, 457.0, 390.0, 454.0, 403.0, 444.0, 422.0, 475.0, 448.0, 410.0, 462.0, 421.0, 418.0, 436.0, 406.0, 401.0, 414.0, 435.0, 371.0, 455.0, 438.0, 421.0, 438.0, 424.0, 422.0, 408.0, 430.0, 382.0, 366.0, 466.0, 424.0, 405.0, 394.0, 407.0, 429.0, 443.0, 458.0, 439.0, 453.0, 403.0, 430.0, 370.0, 393.0, 396.0, 447.0, 441.0, 426.0, 401.0, 381.0, 454.0, 415.0, 474.0, 455.0, 456.0, 406.0, 457.0, 436.0, 399.0, 394.0, 412.0, 383.0, 382.0, 423.0, 417.0, 426.0, 419.0, 414.0, 412.0, 349.0, 407.0, 397.0, 454.0, 415.0, 383.0, 398.0, 411.0, 378.0, 425.0, 423.0, 283.0, 288.0, 290.0, 275.0, 271.0, 293.0, 264.0, 261.0, 260.0, 251.0, 266.0, 383.0, 446.0, 410.0, 367.0, 487.0, 411.0, 392.0, 370.0, 414.0, 411.0, 424.0, 374.0, 427.0, 439.0, 425.0, 432.0, 425.0, 383.0, 388.0, 391.0, 367.0, 441.0, 476.0, 414.0, 491.0, 444.0, 467.0, 431.0, 459.0, 433.0, 439.0, 438.0, 431.0, 396.0, 419.0, 452.0, 468.0, 436.0, 476.0, 445.0, 455.0, 428.0, 430.0, 397.0, 443.0, 411.0, 446.0, 508.0, 428.0, 409.0, 280.0, 327.0, 327.0, 321.0, 282.0, 288.0, 304.0, 286.0, 292.0, 293.0, 306.0, 319.0, 287.0, 309.0, 298.0, 276.0, 271.0, 300.0, 299.0, 292.0, 318.0, 295.0, 315.0, 307.0, 277.0, 286.0, 289.0, 310.0, 289.0, 292.0, 312.0, 502.0, 455.0, 460.0, 422.0, 392.0, 395.0, 370.0, 380.0, 451.0, 409.0, 428.0, 429.0, 429.0, 482.0, 444.0, 456.0, 423.0, 446.0, 465.0, 384.0, 437.0, 487.0, 421.0, 396.0, 432.0, 400.0, 394.0, 397.0, 434.0, 475.0, 467.0, 393.0, 410.0, 425.0, 408.0, 424.0, 418.0, 473.0, 432.0, 459.0, 417.0, 475.0, 433.0, 460.0, 470.0, 423.0, 414.0, 404.0, 453.0, 478.0, 389.0, 448.0, 429.0, 438.0, 381.0, 432.0, 404.0, 445.0, 421.0, 460.0, 440.0, 427.0, 386.0, 443.0, 447.0, 403.0, 439.0, 437.0, 411.0, 376.0, 495.0, 430.0, 412.0, 459.0, 457.0, 318.0, 292.0, 308.0, 299.0, 302.0, 276.0, 274.0, 294.0, 302.0, 295.0, 270.0, 333.0, 285.0, 291.0, 292.0, 288.0, 304.0, 262.0, 289.0, 277.0, 299.0, 293.0, 313.0, 291.0, 337.0, 285.0, 288.0, 291.0, 297.0, 285.0, 305.0, 297.0, 283.0, 287.0, 281.0, 312.0, 356.0, 282.0, 301.0, 301.0, 300.0, 276.0, 292.0, 281.0, 299.0, 316.0, 275.0, 306.0, 302.0, 296.0, 280.0, 283.0, 294.0, 303.0, 290.0, 309.0, 301.0, 296.0, 287.0, 279.0, 281.0, 277.0, 318.0, 292.0, 286.0, 319.0, 289.0, 329.0, 290.0, 300.0, 297.0, 300.0, 296.0, 300.0, 288.0, 322.0, 292.0, 290.0, 277.0, 294.0, 279.0, 286.0, 291.0, 298.0, 273.0, 276.0, 302.0, 308.0, 292.0, 327.0, 297.0, 276.0, 319.0, 277.0, 283.0, 279.0, 289.0, 315.0, 277.0, 281.0, 283.0, 274.0, 290.0, 377.0, 519.0, 406.0, 508.0, 405.0, 426.0, 440.0, 411.0, 411.0, 428.0, 423.0, 417.0, 428.0, 425.0, 401.0, 451.0, 435.0, 414.0, 409.0, 446.0, 413.0, 433.0, 397.0, 447.0, 456.0, 412.0, 412.0, 423.0, 445.0, 449.0, 414.0, 420.0, 419.0, 422.0, 408.0, 436.0, 422.0, 421.0, 468.0, 435.0, 464.0, 406.0, 282.0, 294.0, 305.0, 278.0, 291.0, 272.0, 282.0, 284.0, 333.0, 304.0, 284.0, 290.0, 309.0, 293.0, 300.0, 295.0, 280.0, 295.0, 268.0, 293.0, 300.0, 296.0, 279.0, 282.0, 293.0, 290.0, 302.0, 308.0, 300.0, 285.0, 283.0, 273.0, 274.0, 281.0, 316.0, 294.0, 271.0, 301.0, 297.0, 278.0, 305.0, 310.0, 293.0, 284.0, 307.0, 279.0, 309.0, 287.0, 301.0, 289.0, 275.0, 309.0, 289.0, 290.0, 322.0, 276.0, 299.0, 274.0, 306.0, 271.0, 288.0, 287.0, 300.0, 276.0, 283.0, 308.0, 344.0, 425.0, 391.0, 423.0, 388.0, 418.0, 392.0, 419.0, 400.0, 445.0, 472.0, 447.0, 444.0, 410.0, 381.0, 457.0, 386.0, 412.0, 438.0, 454.0, 439.0, 424.0, 422.0, 400.0, 461.0, 391.0, 426.0, 459.0, 438.0, 414.0, 451.0, 411.0, 407.0, 419.0, 419.0, 414.0, 431.0, 463.0, 422.0, 451.0, 415.0, 418.0, 449.0, 431.0, 420.0, 454.0, 382.0, 421.0, 396.0, 433.0, 406.0, 406.0, 466.0, 419.0, 444.0, 413.0, 446.0, 412.0, 462.0, 448.0, 458.0, 480.0, 430.0, 407.0, 450.0, 429.0, 409.0, 446.0, 419.0, 415.0, 434.0, 411.0, 419.0, 492.0, 429.0, 446.0, 419.0, 460.0, 422.0, 428.0, 453.0, 460.0, 408.0, 455.0, 408.0, 472.0, 438.0, 450.0, 442.0, 403.0, 431.0, 438.0, 438.0, 431.0, 458.0, 493.0, 425.0, 437.0, 443.0, 473.0, 404.0, 448.0, 423.0, 389.0, 418.0, 433.0, 477.0, 422.0, 397.0, 463.0, 425.0, 428.0, 471.0, 407.0, 448.0, 520.0, 447.0, 413.0, 428.0, 420.0, 427.0, 461.0, 431.0, 399.0, 437.0, 466.0, 449.0, 410.0, 430.0, 445.0, 416.0, 393.0, 419.0, 428.0, 421.0, 406.0, 395.0, 476.0, 403.0, 442.0, 416.0, 418.0, 441.0, 475.0, 445.0, 417.0, 391.0, 419.0, 425.0, 432.0, 424.0, 432.0, 416.0, 424.0, 420.0, 442.0, 457.0, 410.0, 422.0, 442.0, 396.0, 453.0, 414.0, 422.0, 417.0, 437.0, 403.0, 426.0, 406.0, 428.0, 483.0, 414.0, 442.0, 450.0, 419.0, 443.0, 443.0, 434.0, 422.0, 429.0, 425.0, 415.0, 421.0, 415.0, 425.0, 414.0, 396.0, 438.0, 422.0, 393.0, 407.0, 442.0, 424.0, 318.0, 270.0, 288.0, 297.0, 279.0, 309.0, 284.0, 304.0, 298.0, 312.0, 275.0, 273.0, 297.0, 310.0, 265.0, 292.0, 281.0, 273.0, 288.0, 289.0, 296.0, 306.0, 287.0, 304.0, 324.0, 279.0, 291.0, 281.0, 287.0, 271.0, 286.0, 270.0, 280.0, 378.0, 453.0, 418.0, 413.0, 394.0, 436.0, 421.0, 445.0, 397.0, 402.0, 388.0, 415.0, 442.0, 409.0, 464.0, 463.0, 404.0, 446.0, 394.0, 447.0, 455.0, 435.0, 422.0, 427.0, 435.0, 423.0, 418.0, 410.0, 372.0, 391.0, 387.0, 396.0, 412.0, 407.0, 424.0, 422.0, 418.0, 418.0, 413.0, 392.0, 401.0, 450.0, 457.0, 443.0, 433.0, 410.0, 437.0, 452.0, 453.0, 457.0, 388.0, 439.0, 406.0, 433.0, 389.0, 433.0, 406.0, 444.0, 404.0, 397.0, 410.0, 441.0, 407.0, 426.0, 392.0, 413.0, 395.0, 444.0, 433.0, 411.0, 418.0, 449.0, 416.0, 419.0, 426.0, 440.0, 462.0, 450.0, 426.0, 484.0, 407.0, 415.0, 420.0, 436.0, 424.0, 408.0, 448.0, 382.0, 413.0, 440.0, 433.0, 444.0, 434.0, 408.0, 453.0, 430.0, 421.0, 441.0, 438.0, 401.0, 450.0, 460.0, 379.0, 405.0, 454.0, 404.0, 424.0, 411.0, 429.0, 422.0, 438.0, 399.0, 432.0, 468.0, 429.0, 423.0, 408.0, 410.0, 412.0, 436.0, 421.0, 433.0, 469.0, 389.0, 467.0, 431.0, 432.0, 415.0, 418.0, 449.0, 457.0, 416.0, 461.0, 398.0, 442.0, 435.0, 426.0, 445.0, 423.0, 403.0, 418.0, 421.0, 409.0, 412.0, 364.0, 413.0, 410.0, 443.0, 448.0, 389.0, 427.0, 443.0, 447.0, 440.0, 419.0, 468.0, 450.0, 402.0, 436.0, 389.0, 424.0, 411.0, 415.0, 396.0, 401.0, 363.0, 293.0, 275.0, 286.0, 283.0, 294.0, 291.0, 284.0, 291.0, 307.0, 289.0, 272.0, 274.0, 275.0, 296.0, 281.0, 290.0, 307.0, 297.0, 305.0, 297.0, 298.0, 292.0, 299.0, 317.0, 276.0, 292.0, 301.0, 312.0, 291.0, 291.0, 290.0, 302.0, 277.0, 280.0, 305.0, 311.0, 269.0, 272.0, 285.0, 308.0, 270.0, 278.0, 299.0, 267.0, 286.0, 276.0, 313.0, 276.0, 287.0, 295.0, 298.0, 291.0, 299.0, 280.0, 295.0, 269.0, 422.0, 422.0, 452.0, 427.0, 447.0, 441.0, 417.0, 453.0, 418.0, 422.0, 448.0, 436.0, 407.0, 426.0, 420.0, 398.0, 460.0, 426.0, 472.0, 421.0, 423.0, 431.0, 429.0, 444.0, 394.0, 401.0, 427.0, 390.0, 436.0, 398.0, 470.0, 348.0, 474.0, 447.0, 376.0, 465.0, 425.0, 372.0, 404.0, 392.0, 458.0, 446.0, 449.0, 435.0, 437.0, 463.0, 410.0, 449.0, 435.0, 442.0, 438.0, 432.0, 395.0, 459.0, 448.0, 461.0, 445.0, 434.0, 439.0, 505.0, 479.0, 474.0, 520.0, 440.0, 457.0, 447.0, 453.0, 497.0, 470.0, 452.0, 470.0, 471.0, 445.0, 469.0, 414.0, 469.0, 486.0, 445.0, 447.0, 479.0, 465.0, 466.0, 449.0, 461.0, 418.0, 435.0, 454.0, 451.0, 390.0, 428.0, 278.0, 281.0, 262.0, 294.0, 290.0, 300.0, 301.0, 305.0, 282.0, 286.0, 299.0, 306.0, 284.0, 265.0, 274.0, 296.0, 298.0, 270.0, 282.0, 400.0, 440.0, 434.0, 466.0, 377.0, 409.0, 423.0, 438.0, 409.0, 423.0, 464.0, 428.0, 408.0, 419.0, 456.0, 412.0, 404.0, 452.0, 466.0, 436.0, 463.0, 454.0, 437.0, 417.0, 432.0, 441.0, 415.0, 453.0, 396.0, 410.0, 434.0, 453.0, 406.0, 429.0, 431.0, 418.0, 407.0, 394.0, 392.0, 480.0, 420.0, 414.0, 422.0, 434.0, 391.0, 440.0, 477.0, 391.0, 444.0, 462.0, 447.0, 453.0, 414.0, 385.0, 416.0, 463.0, 460.0, 359.0, 301.0, 279.0, 281.0, 288.0, 281.0, 305.0, 261.0, 281.0, 285.0, 333.0, 467.0, 434.0, 464.0, 420.0, 375.0, 427.0, 456.0, 408.0, 472.0, 435.0, 439.0, 440.0, 435.0, 410.0, 433.0, 457.0, 463.0, 378.0, 394.0, 465.0, 397.0, 411.0, 418.0, 462.0, 454.0, 443.0, 452.0, 395.0, 438.0, 445.0, 405.0, 424.0, 400.0, 439.0, 415.0, 414.0, 440.0, 421.0, 358.0, 421.0, 373.0, 403.0, 441.0, 441.0, 398.0, 455.0, 411.0, 402.0, 413.0, 429.0, 387.0, 436.0, 417.0, 393.0, 463.0, 466.0, 397.0, 382.0, 443.0, 385.0, 435.0, 441.0, 418.0, 444.0, 424.0, 462.0, 437.0, 436.0, 422.0, 428.0, 434.0, 415.0, 453.0, 452.0, 417.0, 400.0, 401.0, 459.0, 452.0, 395.0, 399.0, 420.0, 410.0, 454.0, 414.0, 398.0, 450.0, 430.0, 398.0, 447.0, 436.0, 446.0, 385.0, 431.0, 409.0, 432.0, 441.0, 412.0, 425.0, 401.0, 411.0, 421.0, 402.0, 449.0, 432.0, 436.0, 424.0, 450.0, 434.0, 473.0, 471.0, 421.0, 460.0, 430.0, 441.0, 413.0, 399.0, 432.0, 396.0, 404.0, 430.0, 391.0, 399.0, 388.0, 445.0, 368.0, 420.0, 427.0, 418.0, 420.0, 382.0, 438.0, 418.0, 386.0, 463.0, 408.0, 397.0, 449.0, 402.0, 425.0, 401.0, 437.0, 398.0, 459.0, 392.0, 395.0, 488.0, 357.0, 396.0, 408.0, 392.0, 411.0, 422.0, 399.0, 413.0, 412.0, 427.0, 421.0, 403.0, 404.0, 447.0, 432.0, 404.0, 416.0, 414.0, 426.0, 428.0, 413.0, 422.0, 422.0, 415.0, 433.0, 415.0, 434.0, 407.0, 433.0, 406.0, 441.0, 457.0, 388.0, 390.0, 413.0, 428.0, 424.0, 429.0, 410.0, 391.0, 466.0, 446.0, 393.0, 409.0, 423.0, 430.0, 428.0, 443.0, 358.0, 278.0, 284.0, 286.0, 294.0, 292.0, 283.0, 279.0, 277.0, 279.0, 286.0, 290.0, 282.0, 312.0, 316.0, 300.0, 406.0, 420.0, 406.0, 470.0, 429.0, 430.0, 450.0, 435.0, 437.0, 392.0, 423.0, 455.0, 419.0, 428.0, 451.0, 416.0, 441.0, 412.0, 423.0, 431.0, 433.0, 421.0, 421.0, 445.0, 417.0, 398.0, 390.0, 408.0, 418.0, 426.0, 399.0, 390.0, 392.0, 395.0, 421.0, 408.0, 394.0, 393.0, 431.0, 407.0, 374.0, 447.0, 431.0, 448.0, 389.0, 442.0, 414.0, 412.0, 397.0, 442.0, 417.0, 420.0, 455.0, 416.0, 404.0, 391.0, 401.0, 415.0, 413.0, 429.0, 391.0, 419.0, 411.0, 413.0, 405.0, 414.0, 428.0, 409.0, 401.0, 388.0, 415.0, 389.0, 467.0, 428.0, 396.0, 426.0, 397.0, 454.0, 407.0, 439.0, 397.0, 430.0, 425.0, 405.0, 404.0, 453.0, 402.0, 424.0, 383.0, 378.0, 383.0, 407.0, 414.0, 425.0, 430.0, 414.0, 448.0, 417.0, 369.0, 417.0, 404.0, 424.0, 399.0, 424.0, 438.0, 429.0, 401.0, 409.0, 410.0, 396.0, 443.0, 429.0, 431.0, 454.0, 403.0, 419.0, 392.0, 380.0, 399.0, 441.0, 439.0, 477.0, 449.0, 514.0, 447.0, 425.0, 425.0, 446.0, 444.0, 430.0, 426.0, 488.0, 459.0, 430.0, 434.0, 391.0, 402.0, 405.0, 450.0, 411.0, 387.0, 432.0, 399.0, 456.0, 428.0, 411.0, 438.0, 417.0, 411.0, 441.0, 394.0, 413.0, 415.0, 413.0, 424.0, 411.0, 400.0, 397.0, 406.0, 433.0, 457.0, 440.0, 448.0, 391.0, 392.0, 414.0, 390.0, 439.0, 425.0, 392.0, 435.0, 399.0, 427.0, 419.0, 455.0, 388.0, 397.0, 463.0, 372.0, 449.0, 416.0, 404.0, 396.0]

Many thanks in advance!

Answer 1

You can add a further step to the processing and detect local maxima of the convolved data instead of just the single maximum point.

One way is to use scipy.signal.find_peaks

By building on to the convolution method, you can add the find_peaks step as shown below:

import numpy as np
from scipy import signal
from matplotlib import pyplot as plt

d = [ your data ]

# Convolution part
dary = np.array(d)
dary -= np.average(dary)

step = np.hstack((np.ones(len(dary)), -1*np.ones(len(dary))))

dary_step = np.convolve(dary, step, mode='valid')

# Get the peaks of the convolution
peaks = signal.find_peaks(dary_step, width=20)[0]

# plots
plt.figure()

plt.plot(dary)

plt.plot(dary_step/10)

for ii in range(len(peaks)):
    plt.plot((peaks[ii], peaks[ii]), (-1500, 1500), 'r')

plt.show()

This renders the output:

The above method finds the upward steps, while you could find the downward steps by simply use scipy.signal.find_peaks(-dary_step) — as suggested in the comments.