I have a kafka streams App which runs on a pod in Kube cluster.
Trying to find a way for Kube to scale up/down the pods based on kafka topic lag.
Has anyone walked the path before and willing to share some details ?.
For Scaling up based on the lag, your app should be able to publish metrics (Using micrometer for example) use meter type (Gauge https://micrometer.io/docs/concepts#_gauges)
Then you will need to collect the metrics using monitoring system (prometheus for example)
scale up/down based on the value of the meter/metric.
(Check this sample project as example https://github.com/learnk8s/spring-boot-k8s-hpa)
Note: But consider my comment as the max number of allowed consumers >= number of topic partitions