Search code examples
prometheusstatsd

StatsD-like counter behaviour in Prometheus


The StatsD "counter" metric type is described as:

gorets:1|c This is a simple counter. Add 1 to the "gorets" bucket. At each flush the current count is sent and reset to 0. If the count at flush is 0 then you can opt to send no metric at all for this counter, by setting config.deleteCounters (applies only to graphite backend). Statsd will send both the rate as well as the count at each flush.

Implementations differ, but the principle is the same:

At each flush the current count is sent and reset to 0.

Emitting this to a compatible back-end, and graphing it with any compatible tooling (Grafana, for example) shows a graph that looks like this:

3 ┃
2 ┃
1 ┃    █
0 ┗━━━━━━━━━━━━━━

In words, the counted value will exist in the "bucket" in which you emit the counter and return to zero immediately afterwards.

With Prometheus there seems only to be counters (which are persistent, and cannot be decremented) and gauges. If a Prometheus counter is incremented it will read the new high-watermark value until the process is restarted.

Prometheus also offers Gauges, which can have their values raised and lowered, however there are some situations where no suitable "negative" action exists, and there's no way to decrement a gauge back to zero after the fact.

Prometheus offers a rate() function which would seem to imply that when run over a counter one can take the rate which should show only the positive changes (no negative changes are possible).

I'd prefer to avoid having to add rate() modifiers to all my graphs.

What is the Prometheus idiomatic approach to most closely emulate the StatsD counter behavior?


Solution

  • The Prometheus equivalent is a Counter. The big difference is that the state lives in memory inside your application, rather than over the network in StatsD. Using rate() on all counters is normal in Prometheus.

    There's more detail on the nitty gritty in this talk.