Does apache storm KafkaSpout
have notion of setting priority among source topics that it is subscribing to?
For instance, if KafkaSpout
is subscribing to topics t1
, t2
, t3
is there a way to control the consumption rate like 70%
from t1
, 20%
from t2
and 10%
from t3
and if there aren't any messages left in topic t1
then allocate t1
's resources to t2
and so on?
No, this is not built in. Kafka doesn't have a concept of topic priority. It's possible to build something like this using the KafkaConsumer.pause
and KafkaConsumer.resume
methods. The spout doesn't have support for this right now, but if you'd like to see it added, you can raise an issue at https://issues.apache.org/jira. If you'd like to work on adding it, you can find the Storm code at https://github.com/apache/storm.