Search code examples
prometheusgrafanapromql

PromQL for each element in module


I have a prometheus exporter, which gives the following result:

# HELP apartment 
# TYPE apartment gauge
apartment{module="ddcga"} 2.323522e+09
# HELP bar 
# TYPE bar gauge
bar{module="ddcga"} 7.50631e+08
[...]

Now i want to see those values in Grafana, but for them to be actually useful, i need to use the delta function.

For one value, this is pretty straight forward: delta(apartment{module="ddcga"}[1h])

But there actually are a lot of values, and adding / changing them all manually is a lot of work and error prone looking forward.

I can already display every value using just {module="ddcga"} but using delta({module="ddcga"}[1h]) gives the error execution: vector cannot contain metrics with the same labelset.

How can i make this work?

Additional:


Solution

  • Prometheus drops metric names from the input time series when it applies various functions such as delta(). This results in the vector cannot contain metrics with the same labelset error, since multiple input time series will have the same set of label=value pairs after metric name removal.

    This task can be resolved with a hack on Prometheus subqueries combined with label_replace() function. For example, the following query would calculate the delta over the previous hour per each time series with the module="ddcgq" label with a minute precision. The inner label_replace() moves the metric name to a metric_name label:

    delta(
      label_replace({module="ddcga"}, "metric_name", "$1", "__name__", "(.+)")[1h:1m]
    )
    

    P.S. This task is much easier to solve with VictoriaMetrics - Prometheus-like monitoring solution I work on. It supports keep_metric_names modifier specifically for this case:

    delta({module="ddcga"}[1h]) keep_metric_names
    

    This query returns one-hour deltas per each metric with the label module="ddcga", while keeping the original metric names untouched. See more details in MetricsQL docs.