Search code examples
prometheusmonitoring

How to query the last value across different pods

I have a service to collect projects' test coverage data from their pull-requests, it was put into different pods internally by the collector. So the data will be like:

example input data:

test_coverage{project="a", pod="pod-a"} 80@1684921872
                                        82@1684921882
test_coverage{project="a", pod="pod-b"} 90@1684982920

How can I get the last coverage value across different pods, so that I can get the value 90.

I tried last_over_time(test_coverage{project="a"}[1w]) but get 2 results with

test_coverage{project="a", pod="pod-a"} 82
test_coverage{project="a", pod="pod-b"} 90

My expectation is only the 90 be returned.

Solution

  • You need to do it in two steps:

    1. Remove pod label. It can be done with aggregation by/without. For your example I'd use max by (project) (test_coverage{project="a"})
    2. Take last value of aggregation with last_over_time. Notice that since argument is not a instant selector you need ti use subquery syntax.

    Resulting query would look like this:

    last_over_time((max by (project) (test_coverage{project="a"}))[1w:])