Search code examples
akkaprometheuskamon

Kamon Prometheus takes long to refresh

I started using Kamon instrumentation recently and facing issues with the rate of the kamon/prometheus http endpoint refresh

Preface:

  • using "io.kamon" %% "kamon-bundle" % "2.1.4" && "io.kamon" %% "kamon-prometheus" % "2.1.4"
  • exposing metrics as http endpoint so that prometheus scrapes them and evaluates every 1 sec
  • created custom Counter, Gauge and Histogram metrics and they are updated 2-3K times per sec inside the Akka actor processing incoming messages
  • The reason to use Kamon instead of standard prometheus client is to get thread safety
  • There is configuration kamon.metric.tick-interval 1 second & kamon.prometheus.refresh-interval 1 second related to the rate of refresh

Problem:

Custom metrics that are exposed at the endpoint (localhost:9095) are not refreshed every second. Approximately, they are refreshed every 60 seconds.

It's not prometheus configuration problem, I'm checking the values on the http endpoint exposed by kamon, manually refreshing the page

Solution

  • This was misconfiguration issue. If you are getting same problem, please make sure that the kamon configuration is on the top level of the application.conf, not inside akka {..} as I had it