We have a Java based kafka streams application which is deployed to a kubernetes cluster using helm charts. For scaling up the deployment we are using the Kubernetes option of replica sets which will run multiple instances of the app. But running multiple instances of the app keeps crashing the pods with the error below.
Exception in thread "****-StreamThread-1" org.apache.kafka.common.errors.FencedInstanceIdException: The broker rejected this static consumer since another consumer with the same group.instance.id has registered with a different member.id.
Too much of your config is static. As per documentation:
group.instance.id
A unique identifier of the consumer instance provided by the end user. Only non-empty strings are permitted. If set, the consumer is treated as a static member, which means that only one instance with this ID is allowed in the consumer group at any time.
It looks like different instances of your application should have different group.instance.id.
In the end, it's not related to k8, helm or anything - and you should be capable of replicating it by starting multiple instances of your application on your local machine, all of them pointing to the same Kafka installation.