Search code examples
grafanapromqlprometheus-pushgateway

PromQL, Grafana - Count how many times metric's labels values has changed

I'm trying to count how many times Metric_x changed it's labels.

In Prometheus I have 4 metrics: Metric_1, Metric_2, Metric_3, Metric_4.

If I use the query {__name__=~"Metric_\d"}, I get the following output:

In table:

Metric_1{buildURL="job/Onboarding_Test/2/", job="test_test", line="22"} 1
Metric_2{buildURL="job/Onboarding_Test/2/", job="test_test", line="33"} 1
Metric_3{buildURL="job/Onboarding_Test/7/", job="test_test", line="33"} 1 
Metric_4{buildURL="job/Onboarding_Test/2/", job="test_test", line="11"} 1

In Graph's legend:

Metric_1{buildURL="job/Onboarding_Test/1/", job="test_test", line="33"}
Metric_1{buildURL="job/Onboarding_Test/2/", job="test_test", line="22"}
Metric_2{buildURL="job/Onboarding_Test/1/", job="test_test", line="11"}
Metric_2{buildURL="job/Onboarding_Test/2/", job="test_test", line="33"}
Metric_2{buildURL="job/Onboarding_Test/3/", job="test_test", line="22"}
Metric_3{buildURL="job/Onboarding_Test/2/", job="test_test", line="11"}
Metric_3{buildURL="job/Onboarding_Test/6/", job="test_test", line="11"}
Metric_3{buildURL="job/Onboarding_Test/7/", job="test_test", line="33"}
Metric_4{buildURL="job/Onboarding_Test/2/", job="test_test", line="11"}

I want to visualise in Grafana (table view) how many times Metrics changes their labels values. (Changes I can see in "Graph tab" in Prometheus)

Expected output:

name val
Metric_1 2
Metric_2 3
Metric_3 3
Metric_4 1
Solution

  • You can use combination of count and last_over_time:

    count(last_over_time({__name__=~"Metric_\d"}[1d])) by (__name__)

    Here:

    • {__name__=~"Metric_\d"} returns all metrics with name Metric_ followed by digit. It's regex selector, if you'll need more metrics, modify accordingly,
    • last_over_time( ... [1d]) takes last seen value during the day (24h). So even if time series marked as stale, and stopped being scraped (or in case of pushgateway, stopped being pushed) its last seen value will be returned. You can set content of [..] according to your needs,
    • count( ... ) return number of time series return by inner query.