I am trying to debug a performance issue related to kafka streams stateful application(We use processor API only). The application queries a number of statestores (close to 55). So, in order to find out the reason for slow processing, i was looking for metrics. I found that we can enable jmx metrics for rocksdb in debug mode.
I have set "metrics.recording.level" to debug. But, for some reason, the application is not reporting any metrics. I started getting task level metrics ( which means that the level is debug). Do i need to enable anything else ?
Also, the application runs as a kubernetes pod and creates the state (from changelog) on deployment/restart.
Kafka version : 2.4.1 Kafka client version : 2.4.1
Am I hitting this https://issues.apache.org/jira/browse/KAFKA-9355
Also, are there any native tools which can be used to directly query rocksdb on application pod. btw, i can exec into the pod for any debugging.
I think you are hitting: https://issues.apache.org/jira/browse/KAFKA-9675
It's already fixed but the fix is not released yet.