Search code examples
apache-kafkakafka-consumer-apiavro

I get message from client with python but there is no message on server with python?

when I write this code and run it on my desktop it works.

from kafka import KafkaConsumer

consumer = KafkaConsumer('ersin_test',
                         group_id='eg61-group',
                         auto_offset_reset='earliest',
                         enable_auto_commit=False,
                         consumer_timeout_ms=10000,
                         bootstrap_servers=['10.1.2.3:9092'])

for message in consumer:
    print(message)

but when I move this code to different machine (server) it does not get any messages from kafka. I run this code like this : python my_script.py

my server has connect kafka server :

telnet 10.1.2.3 9092


Trying 10.1.2.3...
Connected to 10.1.2.3.
Escape character is '^]'.

when I run this code on server it works and gets topic-names from kafka

import kafka
consumer = kafka.KafkaConsumer(group_id='eg61-group',bootstrap_servers=['10.1.2.3:9092'])
t=consumer.topics()
print(t)

{'ersin_test', 'second-topic', 'abc', 'ESB', 'first-topic', 'ersin-topic'}

So what is the problem I cant understand.

how can I solve this?

thanks in advance.

Solution

  • The broker has advertised.listeners set to myname, but this DNS hostname is not resolvable from your client machine. This means that whilst the initial bootstrap connection to the IP address succeeds, the broker returns myname as the hostname on which the client should issue consume/produce requests, and since the client cannot resolve myname it fails.

    You should fix your DNS so that you client machine can correctly resolve myname. If that's not an option then reconfigure your Kafka broker to set the advertised.listeners to the IP address (i.e. advertised.listeners=PLAINTEXT://10.1.2.3:9092)

    Ref: https://rmoff.net/2018/08/02/kafka-listeners-explained/