No messages displayed by consumer on running producer and consumer python scripts

Question

I set up Apache Kafka on an Ubuntu server and tested it by following through first five steps mentioned in https://kafka.apache.org/quickstart and things worked fine.

I then proceeded with the installation of kafka-python 1.4.6 to test in python and wrote simple producer and consumer scripts.

My configuration for listeners
listeners=PLAINTEXT://localhost:9092
advertised.listeners=PLAINTEXT://localhost:9092

Here are the scripts

consum.py

from kafka import KafkaConsumer

consumer = KafkaConsumer('my-topic')
for message in consumer:
        print (message)

prod.py

from kafka import KafkaProducer
from kafka.errors import KafkaError

producer = KafkaProducer(bootstrap_servers='localhost:9092',api_version=(0, 10, 1))

producer.send('my-topic', b'Hello')

On running the scripts the producer script completes immediately and no message is printed by consumer script

Any ideas what I might be missing here.
Thanks

Answer 1

The issue is that your producer script finishes before the message is sent. After running send(), Python reaches the last line of your script and stops running immediately.

You need to call flush() to force the runtime to wait for all messages to be sent before terminating.

from kafka import KafkaProducer
from kafka.errors import KafkaError

producer = KafkaProducer(bootstrap_servers='localhost:9092',api_version=(0, 10, 1))

producer.send('my-topic', b'Hello')

producer.flush()

Apart from that your logic looks fine (and worked for me).