Search code examples
group-byprometheusgaugeprometheus-alertmanager

How to do average of gauge metric in Prometheus

I have gauge metric (sample below). And I want to find the average of metric of last 10 hours for each hour. Not combine of 10 hours. I can easily do in SQl by hour in group by clause. But do not have good idea in prometheus query

{group="PrometheusDemo",resource="FinanceServicesGo",service="analyticsA",warning="1000"} 6
{group="PrometheusDemo",resource="FinanceServicesGo",service="analyticsB",warning="3000"} 9
{group="PrometheusDemo",resource="FinanceServicesGo",service="analyticsC",warning="2000"} 8
...
....
...

I tried below query -

avg({__name__="metricA"}) by (group, service)

Edited question
Problem statement

I have a metric A, with time and value (see image below). In hourly avg column, I took the average of each hours. and then in avg over avg column I took the avg of previous averaged column. I got value 9.12 but If I take the combine average of last 2 hour I will get 8.1. I want avg over avg value (9.12) using prometheus query. How I can do this by using prometheus query?

enter image description here

Solution

  • You're looking for subqueries:

    avg_over_time(avg by (group, service) (avg_over_time({__name__="metricA"}[1h]))[10h:1h])

    The outermost query avg_over_time(query[10h:1h]) will evaluate a time period for 10h back, execute the query at 1h interval and then average those 10 results for each time series.

    The inner query avg by (group, service) (avg_over_time({__name__="metricA"}[1h])) will run 10 times. avg_over_time({__name__="metricA"}[1h]) query will average each initial time series over 1h then get averaged by group and service by avg by (group, service) ().