Does PromQL offer a way to filter metrics by comparing one label value to another label value for the same metric?
Like (hypothetical, is invalid syntax) some_metric{label_a!=label_b}
. Or a way of saying some_metric{label_a} unless some_metric{label_b}
that preserves labels on the left side.
I'm trying to find metrics by the relative value of two different labels on the same metric.
Consider some metric replication_lag
with labels source_region
and region
. The source_region
is where the service is pulling replication data from, the region
is the service instance's location.
I want to find replication_lag
only for cross region replication links, those where source_region
!= region
. In other words I want to filter out those metrics where region
== source_region
.
Is there any way to do this in PromQL?
I've been unable to find any way by which a filter on one label's value can use the value of another label to perform a simple set-filtering operation.
Another way to do this would be to label-match two sets replication_lag{some_unique_key,region}
and replication_lag{some_unique_key,source_region}
- but this would require a means to label-match on an inequality condition for region
and some_region
i.e. perform an anti-join, which PromQL also does not appear to support.
The unless
operator looks like it's close, but it requires exactly matching label sets. I need to preserve both region
and source_region
in the resulting set.
The best I've been able to come up with has been to use label rewriting and the unless
operator to generate one label that can be used for set-subtraction.
some_metric{label_a!=label_b}
might be expressed as
label_join(some_metric{}, "join_key", ",", "label_a")
unless on (join_key)
label_join(some_metric{role="replica"}, "join_key", ",", "label_b")
It's verbose, cumbersome, probably inefficient, and it produces an unwanted join_key
label that PromQL lacks a label_drop
function to get rid of. There must be a better way.
Prometheus doesn't provide functions for comparing values across different labels the same metrics.
If you still need this functionality, you can try using an alternative Prometheus-like solution I work on, which supports such function - labels_equal. For example, the following MetricsQL query returns some_metric
time series where region
label doesn't equal source_region
label:
some_metric unless labels_equal(some_metric, "region", "source_region")
This query uses unless
operator for filtering out series with identical values for region
and source_region
labels.