Search code examples
apache-kafkaapache-storm

storm-kafka KafkaSpout: setting priority among topics


Does apache storm KafkaSpout have notion of setting priority among source topics that it is subscribing to?

For instance, if KafkaSpout is subscribing to topics t1, t2, t3 is there a way to control the consumption rate like 70% from t1, 20% from t2 and 10% from t3 and if there aren't any messages left in topic t1 then allocate t1's resources to t2 and so on?


Solution

  • No, this is not built in. Kafka doesn't have a concept of topic priority. It's possible to build something like this using the KafkaConsumer.pause and KafkaConsumer.resume methods. The spout doesn't have support for this right now, but if you'd like to see it added, you can raise an issue at https://issues.apache.org/jira. If you'd like to work on adding it, you can find the Storm code at https://github.com/apache/storm.