I currently use a prometheus alertmanager alert to determine the number of boxes that are running at a given time.
I would like to create an alert that will alarm if the number of online boxes changes. I use something like this:
sum(up{app="mybox"}) != sum(up{app="mybox"} offset 5m)
This will detect a state change, but it does not report the current / previous value. All $labels.value will return is a true/false. Is there a way to write this query so that A) it both alarms on change, and B) reports the different values?
As seen in alert templating, there is only one $value available which is the expression of the rule.
In you case, it will be the new value of sum(up{app="mybox"}). Depending on your use case, you could express it as
sum(up{app="mybox"}) - sum(up{app="mybox"} offset 5m) != 01 - clamp_max(sum(up) / sum(up{app="mybox"} offset 5m), 0) != 0If you really need the other (previous) value, you can use a query template but don't put it in a label. Otherwise, a different alert will be generated for each previous value.
expr: sum(up{app="mybox"}) != sum(up{app="mybox"} offset 5m)
annotations:
description: The new value {{ $value }} is different from the previous: {{with query "sum(up{app='mybox'} offset 5m)"}}{{ . | first | value}}{{end}}.
Note: the query may be simplified in your case but I wrote the general case.