I have a prometheus exporter, which gives the following result:
# HELP apartment
# TYPE apartment gauge
apartment{module="ddcga"} 2.323522e+09
# HELP bar
# TYPE bar gauge
bar{module="ddcga"} 7.50631e+08
[...]
Now i want to see those values in Grafana, but for them to be actually useful, i need to use the delta
function.
For one value, this is pretty straight forward: delta(apartment{module="ddcga"}[1h])
But there actually are a lot of values, and adding / changing them all manually is a lot of work and error prone looking forward.
I can already display every value using just {module="ddcga"}
but using delta({module="ddcga"}[1h])
gives the error execution: vector cannot contain metrics with the same labelset
.
How can i make this work?
Additional:
{{__name__}}
as legendPrometheus drops metric names from the input time series when it applies various functions such as delta(). This results in the vector cannot contain metrics with the same labelset
error, since multiple input time series will have the same set of label=value
pairs after metric name removal.
This task can be resolved with a hack on Prometheus subqueries combined with label_replace() function. For example, the following query would calculate the delta over the previous hour per each time series with the module="ddcgq"
label with a minute precision. The inner label_replace()
moves the metric name to a metric_name
label:
delta(
label_replace({module="ddcga"}, "metric_name", "$1", "__name__", "(.+)")[1h:1m]
)
P.S. This task is much easier to solve with VictoriaMetrics - Prometheus-like monitoring solution I work on. It supports keep_metric_names
modifier specifically for this case:
delta({module="ddcga"}[1h]) keep_metric_names
This query returns one-hour deltas per each metric with the label module="ddcga"
, while keeping the original metric names untouched. See more details in MetricsQL docs.