Prometheus alert on metric value increase above threshold

Question

I try to create expr for Prometheus AlertManager to fire alert on metric value when value is above treshold (for example, 100) but alert must be fired ONLY if the value (on next scrape) after 1min incereased (140->185), if value drops (135->110), then no alert must be fired.

Here is the graph in the image. Metric ups and downs below threshold should be ignored.

Tried to use avg_over_time with offset, but could not managed to write successful promql expr.

Answer 1

You can use query like this:

my_metric >= (my_metric offset 1m > 100)

Here:

1. my_metric offset 1m > 100 will return value only if value of metric one minute ago was greater than 100,
2. my_metric >= ( .. ) will return value only if current value is greater than value returned on step 1 (and it was returned on step 1).

Adjust 1m to your actual scrape_interval, so that offset queries previous value.

Additionally, remember, that alert rules are evaluated once in evaluation_interval, and if it's lower than scrape_interval, your rule might need adjustment, to check multiple previous values.