Search code examples
alertgrafanadashboard

Grafana Alerts when two hits are registered with 5 min between them

Hi I am trying to setup a Grafana alert and doing a simple alert when a value is below/over a value or a query is "hit" once is easy.

But I can't figure out how to set up an alert that goes off, when a query is hit - and hit again 5-10 min later.

I will try to visualize it with the following drawing:

Illustration of what I mean with a "hit" followed by another hit 5-10 min later

I want a alert to check if there is an "hit" (the red lines). This first "hit" (where the blue arrow is) can be followed by many hits right after within the next 5 minutes, but if it goes again between 5-10 minutes later on (the green arrow), then an alert should be triggered.

So somehow an alert that checks every 5 minutes if there in two 5-minute slots after each other has been a count >= 1. Meaning >= 1 hit pr. slot.

Intuitively I would set it up as the following:

intuitivily setup of Alert in Grafana

Here I evaluate every 5 minute for 5 minute (EDIT should be every 5m for 10m), which is what I want, but the problem here is, that if there is 2 counts/hits in the first 5 minutes and 0 in the following 5 minutes, then the average would still be 1. And also the sum of counts/hits will still be 2. So how do I tell it to check for 1 count in two following 5 minute slots?

Solution

  • Found the answer - or one solution, that is quite simple:

    enter image description here

    By setting two queries:

    1. Tracks the last 5 minutes and registers if count >= 1
    2. Tracks the last 10-5 minutes and registers if count >= 1

    if both conditions are met, then an alert is send.