I'm trying to create an alert expression that fires if the given metric is not the same across all targets. Let's call my metric my_gauge where the expression console output shows:
my_gauge{group="test",instance="huey:9100",job="example"} 10
my_gauge{group="test",instance="duey:9100",job="example"} 10
my_gauge{group="test",instance="luey:9100",job="example"} 7
I'm interested in the target results that are below the highest threshold. So my initial thought was to compare it to the max (my_gauge) result which is:
{} 10
However my_gauge < max(my_gauge) returns no data (the expected result here would be luey). How do I write an expression that would return a target with a divergent metric?
If you only need to alert on wether at least one value is different, you can compare if minimum value is the same as maximum value:
expr: min(my_gauge) by(group) != max(my_gauge) by(group)
If you want to alert on every value that differs, you can use the median to determine the common value (10 in your example). And then compare to it.
expr: my_gauge != on(group) group_left quantile(0.5, my_gauge) by(group)
Regarding your expression, you are missing the vector matches:
expr: my_gauge < on() group_left max(my_gauge)