How to configure Akka to expose metrics to Prometheus using Kamon?

Question

I am trying to configure my project which is based on Akka 2.6.10 to expose metric values to Prometheus. I saw this question which uses Kamon but I couldn't figure out what I am missing on my configuration. My build.sbt file has the following configuration:

name := """explore-akka"""
version := "1.1"
scalaVersion := "2.12.7"
val akkaVersion = "2.6.10"
lazy val kamonVersion = "2.1.9"

libraryDependencies ++= Seq(
  // Akka basics
  "com.typesafe.akka" %% "akka-actor" % akkaVersion,
  "com.typesafe.akka" %% "akka-testkit" % akkaVersion,

  // Metrics: Kamon + Prometheus
  "io.kamon" %% "kamon-core" % kamonVersion,
  "io.kamon" %% "kamon-akka" % kamonVersion,
  "io.kamon" %% "kamon-prometheus" % kamonVersion
)

and the plugins.sbt:

resolvers += Resolver.bintrayRepo("kamon-io", "sbt-plugins")
addSbtPlugin("io.kamon" % "sbt-aspectj-runner" % "1.1.1")

I added on the application.conf:

kamon.instrumentation.akka.filters {
  actors.track {
    includes = [ "CounterSystem/user/Counter**" ]
  }
}

then I start a MainClass which call the counterActor:

object MainClass extends App {
  Kamon.registerModule("akka-test", new PrometheusReporter())
  Kamon.init()
  CounterActor.run()
}

import akka.actor.{Actor, ActorSystem, Props}
import kamon.Kamon

object CounterActor extends App {
  run()

  def run() = {
    import Counter._
    val actorSystem = ActorSystem("CounterSystem")
    val countActor = actorSystem.actorOf(Props[Counter], "Counter")
    (1 to 100).foreach { v =>
      Thread.sleep(1000)
      countActor ! Increment
    }
    (1 to 50).foreach { v =>
      Thread.sleep(1000)
      countActor ! Decrement
    }
    countActor ! Print
  }

  class Counter extends Actor {
    import Counter._
    val counter = Kamon.counter("my-counter")
    var count = 0
    override def receive: Receive = {
      case Increment =>
        count += 1
        println(s"incrementing... $count")
        counter.withoutTags().increment()
      case Decrement =>
        count -= 1
        println(s"decrementing... $count")
        counter.withoutTags().increment()
      case Print =>
        sender() ! count
        println(s"[counter] current count is: $count")
    }
  }
  object Counter {
    case object Increment
    case object Decrement
    case object Print
  }
}

I think that after this I could launch the application using sbt run and listen to the metrics on the Prometheus console (http://127.0.0.1:9090/graph), but I do not see any metrics related to my actor. My guess is that I have to configure the scrape_config at the prometheus file /etc/prometheus/prometheus.yml. Am I right? How should I configure it?

Answer 1

I had to configure Prometheus to scrape the Kamon web service through the config file

cat /etc/prometheus/prometheus.yml

global:
  scrape_interval: 15s
scrape_configs:
  - job_name: "kamon"
    scrape_interval: "5s"
    static_configs:
      - targets: ['localhost:9095']
    metrics_path: /

add these 2 Kamon libraries at build.sbt:

  "io.kamon" %% "kamon-bundle" % "2.1.9",
  "io.kamon" %% "kamon-prometheus" % "2.1.9",

add this configuration at application.conf:

kamon.instrumentation.akka.filters {
  actors.track {
    includes = [ "AkkaQuickStart/user/*" ]
    # excludes = [ "AkkaQuickStart/system/**" ]
  }
}

Start Kamon and call the counter:

Kamon.init()
val counterSendMsg = Kamon.counter("counter-send-msg")
counterSendMsg.withTag("whom", message.whom).increment()

Here is the full demo application from Akka quick start with Kamon configured to count messages:

import akka.actor.typed.scaladsl.Behaviors
import akka.actor.typed.{ActorRef, ActorSystem, Behavior}
import kamon.Kamon

import scala.util.Random

object Greeter {
  val counterSendMsg = Kamon.counter("counter-send-msg")
  def apply(): Behavior[Greet] = Behaviors.receive { (context, message) =>
    context.log.info("Hello {}!", message.whom)
    //#greeter-send-messages
    message.replyTo ! Greeted(message.whom, context.self)
    counterSendMsg.withTag("whom", message.whom).increment()
    //#greeter-send-messages
    Behaviors.same
  }
  final case class Greet(whom: String, replyTo: ActorRef[Greeted])
  final case class Greeted(whom: String, from: ActorRef[Greet])
}

object GreeterBot {

  def apply(max: Int): Behavior[Greeter.Greeted] = {
    bot(0, max)
  }

  private def bot(greetingCounter: Int, max: Int): Behavior[Greeter.Greeted] =
    Behaviors.receive { (context, message) =>
      val n = greetingCounter + 1
      context.log.info("Greeting {} for {}", n, message.whom)
      if (n == max) {
        Behaviors.stopped
      } else {
        message.from ! Greeter.Greet(message.whom, context.self)
        bot(n, max)
      }
    }
}

object GreeterMain {

  def apply(): Behavior[SayHello] =
    Behaviors.setup { context =>
      //#create-actors
      val greeter = context.spawn(Greeter(), "greeter")
      //#create-actors

      Behaviors.receiveMessage { message =>
        //#create-actors
        val replyTo = context.spawn(GreeterBot(max = 3), message.name)
        //#create-actors
        greeter ! Greeter.Greet(message.name, replyTo)
        Behaviors.same
      }
    }
  final case class SayHello(name: String)
}

object AkkaQuickstart {

  def main(args: Array[String]): Unit = {
    run()
  }

  def run() = {
    Kamon.init()
    import GreeterMain._
    val greeterMain: ActorSystem[GreeterMain.SayHello] = ActorSystem(GreeterMain(), "AkkaQuickStart")
    val allPerson = List("Charles", "Bob", "Felipe", "Simone", "Fabio")
    def randomPerson = allPerson(Random.nextInt(allPerson.length))
    while (true) {
      greeterMain ! SayHello(randomPerson)
      Thread.sleep(1000)
    }
  }
}

and my Prometheus web console: