kafka python - What is the right way to find new kafka topics that a consumer has not been yet subscribed to?

Question

I am new to the kafka world and trying to do the following for a kafka consumer in python

1. get a list of all kafka topics.
2. get a list of topics a consumer has subscribed to.
3. subscribe to new topics (that have not yet been subscribed to).

Note: I am ok to use either confluent-kafka / kafka-python library to achieve this.

Any help would be appreciated.

Answer 1

if you created your Consumer with kafka-python

from kafka import KafkaConsumer

consumer = KafkaConsumer(
 bootstrap_servers = 'hostname:port',
)

you can review the list of topics available with

consumer.topics()

when you subscribe to topics, you can review the consumer subscriptions with

consumer.subscription()

You can do one minus the other to find the topics you still need to subscribe to and then you can do so with

consumer.subscribe(topics=[list_of_topic_names])