Search code examples
kubernetesapache-kafka-streams

How do I scale Kafka Streams app running on kubernetes


I have a kafka streams App which runs on a pod in Kube cluster.

Trying to find a way for Kube to scale up/down the pods based on kafka topic lag.

Has anyone walked the path before and willing to share some details ?.


Solution

  • For Scaling up based on the lag, your app should be able to publish metrics (Using micrometer for example) use meter type (Gauge https://micrometer.io/docs/concepts#_gauges)

    Then you will need to collect the metrics using monitoring system (prometheus for example)

    scale up/down based on the value of the meter/metric.

    (Check this sample project as example https://github.com/learnk8s/spring-boot-k8s-hpa)

    Note: But consider my comment as the max number of allowed consumers >= number of topic partitions