Search code examples
strimzi

Is there an automatic KafkaRebalance trigger happens when a new broker added to the cluster?

  1. I have created a kafka cluster using examples/metrics/kafka-cruise-control-metrics.yaml with 3 replicas.
  2. I have added a new broker by changing the replicas to 4 and reapplied.
  3. In order to ensure the partitions are distrubuted to new broker I have run examples/cruise-control/kafka-rebalance-full.yaml with strimzi.io/rebalance-auto-approval: "true"
  4. After that the partitions are rebalanced with all the brokers including the new broker.

Is it possible that Strimzi/CruiseControl automatically detects a new broker being added and trigger a KafkaRebalance without applying a KafkaRebalance yaml with auto approve. This will be helpful since there is no Manual intervention required during scale in/out.

Solution

  • It's not possible. The KafkaRebalance custom resource is the way to interact with Cruise Control using the Kubernetes approach and the Strimzi operator (you could even use the Cruise Control REST API but it's not the recommended way). If you add a new broker, so scaling up through the Kafka custom resource, there is no support in the Strimzi operator to detect the scale up and starting the rebalance. You would need to use a KafkaRebalance resource to do this. Having this feature is on our roadmap, but it's not implemented yet and we don't have an ETA right now.