I have two PromQL queries in Grafana.
Query 1: max_over_time(counter{label="label1"}[5m])
Query 2: max_over_time(counter{label="label1"}[5m] offset 10m)
There’s an exact match between the labels in both queries, so I don’t believe I need to use the on() function.
I would like to compute the difference between these queries…
Query 3: max_over_time(counter{label="label1"}[5m]) - max_over_time(counter{label="label1"}[5m] offset 10m)
Query 3 returns a resulting vector which is correct for the most part. If, for example, the resulting vector of Query 1 has an entry, at the i’th position, with value 1500 and the resulting vector of Query 2 has an entry at the i’th position with value 1000. Then the i’th position of the resulting vector in Query 3 becomes 1500-1000= 500.
But when Query 1 has a value of 1000 and Query 2 a value of null (which is formatted as 0), the result becomes 1000 - null = null.
I would like the result to be 1000 in this case. I have attempted to convert all null values to zero, but based on what I’ve read, Prometheus seems to already treat nulls as zeros. I have also attempted to use vector(0):
(max_over_time(counter{label="label1"}[5m]) or vector(0)) - (max_over_time(counter{label="label1"}[5m] offset 10m) or vector(0))
But this doesn't change the result.
Here is a subset of the results obtained from Grafana's query inspector for all three queries.
Correct results for the difference query
I get correct results for the difference query when there are non-null values. However as soon as null values occur, the difference can no longer be computed.
Incorrect difference values when there are null values. 1000-null should be 1000 not null.
I would really appreciate some helpful tips. Thanks in advance.
Try this query:
sum(max_over_time(counter{label="label1"}[5m]) or vector(0)) - sum(max_over_time(counter{label="label1"}[5m] offset 10m) or vector(0))