Search code examples
apache-kafkaprometheusmetrics

Prometheus - calculating the percentage increase

I'm very new to using Prometheus metrics and I've been asked to write an alert based on whether a value has increased by a 10% over the last five minutes.

To be more concise, I've to report when my Kafka topic deadletters count increases by 10% within 5 minutes.

I can count the current deadletters on the topic/sub using this query:

count by(topic) (kafka_burrow_topic_partition_offset{topic=~".+__deadletter__.+my-subscription-name.+"})

What I now need to do is get the same number from 5 mins ago and then calculate the percentage increase from then and now:

Percentage Increase = [ (Final Value - Starting Value) / |Starting Value| ] × 100

Can anyone give me a steer on how to calculate the percentage difference?

I've seen examples like this but cant get it to work for me: (sum by(topic) (increase(kafka_burrow_topic_partition_offset{topic=~".+__deadletter__.+"}[5m])) > 0)

Solution

  • If you use the following query for calculating the value at the moment:

    count by(topic) (kafka_burrow_topic_partition_offset{topic=~".+__deadletter__.+my-subscription-name.+"})

    then the value 5 minutes ago can be calculated by adding offset 5m to the query:

    count by(topic) (kafka_burrow_topic_partition_offset{topic=~".+__deadletter__.+my-subscription-name.+"} offset 5m)

    Note that the offset modifier must be put immediately after closing curly brace in the query above. See these docs for more details on this.

    Given that, the following query would show the percentage increase for the value during the last 5 minutes:

    (
  count by (topic) (
    kafka_burrow_topic_partition_offset{topic=~".+__deadletter__.+my-subscription-name.+"}
  ) - count by (topic) (
    kafka_burrow_topic_partition_offset{topic=~".+__deadletter__.+my-subscription-name.+"} offset 5m
  )
) / count by (topic) (
  kafka_burrow_topic_partition_offset{topic=~".+__deadletter__.+my-subscription-name.+"} offset 5m
) * 100

    P.S. Note that the query contains multiple repeated parts. These parts can be simplified to the following MetricsQL query using with templates from VictoriaMetrics:

    with (
  q = count by (topic) (
    kafka_burrow_topic_partition_offset{topic=~".+__deadletter__.+my-subscription-name.+"}
  )
)
(q - q offset 5m) / (q offset 5m) * 100