Search code examples
prometheusgrafanapromql

How to avoid "vector cannot contain metrics with the same labelset" error when plotting rate() from several metrics (same labelset, different names)


In Prometheus sometimes we need to plot several metrics at onces (say, having name, fitting the same regex) like that PromQL query:

{name=~"camel_proxy.*count"}

and it works fine, the same labelset lines are plotted with the different names.

When we want to plot the rate() of them, we face the error from the title:

rate({name=~"camel_proxy.*count"}[5m])

So, the way here is to make labelset not the same, and to move the __name__ to some label, making each labelset to be unique:

rate(label_replace({name=~"camel_proxy.*count"},"name_label","$1","name", "(.+)")[5m])

But we are still getting the error like

1:90: parse error: ranges only allowed for vector selectors"

How to avoid it and plot the rates correctly?


Solution

  • The label_replace() inside rate() triggers subquery processing, which may return unexpected results, since the rate() starts working with calculated samples returned from label_replace() instead of raw samples stored in the database.

    MetricsQL at VictoriaMetrics (this is Prometheus-like monitoring solution I work on) provides more elegant solution for vector cannot contain metrics with the same labelset error - keep_metric_names modifier. Just put this modifier after the function, which strips metric names, in order to instruct it to keep metric names:

    rate({name=~"camel_proxy.*count"}[5m]) keep_metric_names
    

    This solution avoids triggering subquery processing, so the rate() function continues working on raw samples stored in the database.