cant read\write from kafka topic

Question

I have Kafka running on a container on my desktop. I can connect to it just fine using a tool called "Kafka tool" where I can see my topics for example.

I'm having issues reading and writing to/from a Kafka topic. what's annoying me is that it won't give me an error message, it's behaving like if the topic doesn't have any messages on it, but it does, I can see using the tool, even added two messages manually.

the topic exists and has two messages on it (which I added manually using this UI)

problem: the code that sends messages to the topic runs fine, but the messages don't make it to Kafka the code that reads messages from the topic doesn't read anything. It sits there like if there are no messages to be read. Also, I can use the same consume to list the topics (which indicates the connection was successful)

The kafka version is 2.4. Any idea what the problem may be? I have tried "bootstrap_servers=['localhost:9092', 'kafka-server:9092']" but it also didnt work

Thanks

Answer 1

1. KafkaProducer: You need to execute flush after send

producer.send('testTopic', b'Hello, World!')
producer.flush()

2. KafkaConsumer: Specify bootstrap_servers and auto_offset_reset

consumer = KafkaConsumer('testTopic',
                         bootstrap_servers=['localhost:9092'],
                         auto_offset_reset='earliest')
for message in consumer:
    print(message)