Search code examples
prometheusgrafanapromql

How correctly display data with a frequency of 15 seconds in Prometheus?

I need to count the number of requests to the backend within 15 seconds. Requests are sent from the client every 15 seconds using javascript (setInterval). I have a metric that counts the number of requests to the backend.

# TYPE backend_counter_total counter
backend_counter_total{project_id="333",} 212.0

Everything works fine when using the 1 minute interval. It displays the 1 minute data correctly with code:

increase(backend_counter_total[1m])

enter image description here

but if the interval is lowered to 15 seconds

increase(backend_counter_total[15s])

enter image description here

the graph becomes broken with zeroing after each step. I can't figure out how to fix the situation and make the graph smooth at 15 seconds interval.

Solution

  • The way increase, rate and irate functions work is by getting the datapoints in the given time interval and then if there are at least two samples in it does an interpolation. This is done at every point evaluated to generate the derived time-series.

    If you are scraping the metrics every 15 seconds, you will have a sample approximately, but not exactly, every 15 seconds. This distinction is important to understand why the chart that you see is inconsistent. When trying to get the derived metric using a 15s interval, prometheus will find some intervals where there are two samples and others when there's only one. In the former you will see data but on the latter you won't as it is not able to extrapolate the value from a single sample.

    When using a 1-minute interval, the odds of getting 2 samples collected within each evaluated interval are really high, thus it generates a consistent time-series.

    This is the reason why prometheus recommends using an interval at least 4 times the scrape interval for any of these derived metrics. Although practically, if you don't get any drops, a 2 or 3 times the scrape interval will yield consistent results.

    The conclusion is that if you need to have 15s resolution rates, you will need to lower your scrape interval to, at least, 5s.