Search code examples
javaapache-kafkaapache-zookeeperapache-storm

apache zookeeper kafka path

I have an application that sets up storm connection to kafka topics. The settings I imitated from another similar repo in our org has a property like the following:

zookeeper.connect=127.0.0.1:2181/kafka_0.9

I've found that with this setting, in my local dev environment, my app throws an error when it tries to create storm spout. The error is traced to a call to getNumPartitions. The detailed error log is the following:

org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /brokers/topics/Moment_2018_01_16_07_59_08/partitions
at org.apache.zookeeper.KeeperException.create(KeeperException.java:111) ~[zookeeper-3.4.6.jar:3.4.6-1569965]
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) ~[zookeeper-3.4.6.jar:3.4.6-1569965]
at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1590) ~[zookeeper-3.4.6.jar:3.4.6-1569965]
at org.apache.curator.framework.imps.GetChildrenBuilderImpl$3.call(GetChildrenBuilderImpl.java:214) ~[curator-framework-2.5.0.jar:na]
at org.apache.curator.framework.imps.GetChildrenBuilderImpl$3.call(GetChildrenBuilderImpl.java:203) ~[curator-framework-2.5.0.jar:na]
at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107) ~[curator-client-2.5.0.jar:na]
at org.apache.curator.framework.imps.GetChildrenBuilderImpl.pathInForeground(GetChildrenBuilderImpl.java:199) ~[curator-framework-2.5.0.jar:na]
at org.apache.curator.framework.imps.GetChildrenBuilderImpl.forPath(GetChildrenBuilderImpl.java:191) ~[curator-framework-2.5.0.jar:na]
at org.apache.curator.framework.imps.GetChildrenBuilderImpl.forPath(GetChildrenBuilderImpl.java:38) ~[curator-framework-2.5.0.jar:na]
at storm.kafka.DynamicBrokersReader.getNumPartitions(DynamicBrokersReader.java:91) ~[storm-kafka-0.9.6.jar:na]

When I take out the trailing /kafka_0.9 from the property like so:

zookeeper.connect=127.0.0.1:2181

Then this error disappears. My speculation is that the code for our org has a specific path /kafka_0.9 setup in our staging & prod environment (probably to specify using kafka 0.9). But I wonder how I can create this path in local dev environment as well just to be consistent with our staging & prod environments? Currently my docker-compose.yml has the following for zookeeper & kafka:

zookeeper:
  image: myorg/zookeeper:3.4.8
  ports:
    - "2181:2181"

kafka:
  image: myorg/kafka:kafka-0.10
  hostname: myapp.docker
  ports:
    - "9092:9092"
  environment:
    EXPOSED_HOST: myapp.docker
    KAFKA_PORT: 9092
    KAFKA_ADVERTISED_PORT: 9092
    ZOOKEEPER_PORT_2181_TCP_ADDR: zookeeper
    ZOOKEEPER_PORT_2181_TCP_PORT: 2181
    ZOOKEEPER_IP: zookeeper
  links:
    - zookeeper
  extra_hosts:
    - "localhost:0.0.0.0"
Solution

  • try add Environment Variables: CHROOT=/kafka_0.9

    Zookeeper also allows you to add a "chroot" path which will make all kafka data for this cluster appear under a particular path. This is a way to setup multiple Kafka clusters or other applications on the same zookeeper cluster. To do this give a connection string in the form hostname1:port1,hostname2:port2,hostname3:port3/chroot/path which would put all this cluster's data under the path /chroot/path. Note that you must create this path yourself prior to starting the broker and consumers must use the same connection string.