Search code examples
pythonapache-kafkakafka-python

Unable to consume messages using python-kafka

I was trying to replicate the steps given in the blog. While trying, there given a Kafka Consumer and Kafka Producer python code, I am able run the code in python interactive terminal, and able the consumer console gives the output, But if I pass them in a python file (*.py), it consumes nothing.

Consumer

from kafka import KafkaConsumer
consumer = KafkaConsumer('sample')
for message in consumer:
    print (message)

Producer

from kafka import KafkaProducer
producer = KafkaProducer(bootstrap_servers='localhost:9092')
producer.send('sample', b'Hello, World!')
producer.send('sample', key=b'message-two', value=b'This is Kafka-Python')

How can I make it work in a python file?

Solution

  • I just added producer.flush() to the producer code, and it started to work.

    Because Kafka clients send messages in batches, not immediately to reduce load on the brokers.

    You didn't send enough data initially for a flush to occur on its own so your data just sat in memory while your app ended.

    Refer batch.size producer property