Search code examples
prometheusgrafanaprometheus-alertmanagergrafana-alerts

Dealing with NoData in Grafana Managed Alerts


I have a expression that has my alerting conditions. If ANY of the conditions returns NoData, it appears to preempt evaluation of the other conditions, even if doing so should prevent evaluation of the condition that returns NoData. Is there any way to avoid this? I've created my conditions, to avoid alerting on certain days and outside of certain hours, which should preempt the evaluation of the final condition check, the one which returns no data on certain days and outside of certain hours. However, the NoData seems to preempt everything else. In the snippet below, the first two conditions are the day of week and hour of day checks and A is the one that returns NoData. Thanks for your help. It makes no difference if I remove that HAS NO VALUE condition. If it has no value during the times allotted, we want it alerting.

enter image description here


Solution

  • First of all, give transformations a try as suggested in the Grafana community. Unfortunately this does not work for alert rules or in panels that define an alert rule.

    Without transformations, things can get complicated.

    You are using "Classic condition". Try using a combination of "Reduce" and "Math" instead.

    Sometimes, when you run out of data, you get NaN as result of a query. I find it useful that the math expression 0 && NaN evaluates to 0 (while NaN > x always returns NaN, which can sometimes be annoying).

    So you might be able to build a query D that returns 0 instead of NaN whenever queries A, B or D return NaN and a positive value in all other cases. You can then use something like:

    D && (2 < C < 6) && (15 < B < 21) && (A < 1)

    Note that I ommited the reduce expressions for simplicity.