Search code examples
javaapache-zookeeperapache-storm

Apache storm Kafka SpoutConfig for Zookeeper quorum

To configure a kafkaSpout, it Takes BrokerHosts which in turn take one zookeeper host. 

BrokerHosts host = new ZkHosts("server-1:2181");

SpoutConfig spoutConfig = new SpoutConfig(host, TopologyConstants.KAFKA_QUEUE.SOURCE,
            "/" + TopologyConstants.KAFKA_QUEUE.SOURCE, ID);

KafkaSpout kafkaSpout = new KafkaSpout(spoutConfig);

Problem is, If i have a zookeeper quorum(cluster of 3 zk servers), how do i configure KafkaSpout to take all the members of quorum instead of only 1. As one zookeeper server may get down and the whole topology will be unavailable.

Solution

  • Found answer to my question: connection string format is "host1:port1,host2:port2,host3:port3..." (storm-kafka uses Curator under the hood), so just supply multiple Zookeeper server urls to the ZkHosts constructor