I have a Prometheus vector metric (etcd_network_client_grpc_received_bytes_total) with a label (instance). The metric has a different value for each of a bunch of label values (i.e. one value per instance). I want to find all instances for which the value is smaller than 70% of the average of all instances.
For example, if the vector had the following values:
etcd_network_client_grpc_received_bytes_total{instance="192.168.0.18:2399"} 19021275139
etcd_network_client_grpc_received_bytes_total{instance="192.168.0.22:2399"} 390020
etcd_network_client_grpc_received_bytes_total{instance="192.168.0.30:2399"} 19021275254
etcd_network_client_grpc_received_bytes_total{instance="192.168.0.48:2399"} 38992
etcd_network_client_grpc_received_bytes_total{instance="192.168.0.49:2399"} 1992
...then the query should return the 2nd, 4th and 5th element.
What doesn't work:
etcd_network_client_grpc_received_bytes_total < (0.7 * avg (etcd_network_client_grpc_received_bytes_total))
(I guess because the whole thing is evaluated for each label value)
Is this possible?
(I should actually be comparing rates, not totals, but let's keep it simple for now)
etcd_network_client_grpc_received_bytes_total
< ignoring(instance) group_left
0.7 * avg without(instance)(etcd_network_client_grpc_received_bytes_total)
See How do I compare select all values larger than their average?