Kafka producer unable to connect to broker via Internet. Works well if in local network as broker. Telnet connectivity working from internet

Question

I have a Kafka broker running on a computer on my home network. When I connect my java producer (on another computer) to this broker from the same wifi network, it works absolutely well and is able to post messages without any issues.

As next steps for my application, I have exposed my computer and Kafka's 9092 port over internet. I have also opened all firewall on my system for this port. And when I do telnet/nc connection attempt from any other network, it connects well. The Kafka host also shows the connection made in the netstat output.

 telnet DNSHOSTNAME 9092
Trying MYIP...
Connected to DNSHOSTNAME.
Escape character is '^]'.

But from this external network, when I use my exact same kafka java producer to send to my home computer from this outside network, it always fails with the following error. I also do not see any connection attempts or errors in the Kafka logs related to this.

    Error while producing message to topic :TP1-0@-1
org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s) for TP1-0: 30005 ms has passed since batch creation plus linger time

Another interesting point is that, I have other apps - RabbitMQ and Tomcat also on the same machine as Kafka with the same firewall and router rules. My same java app is able to publish to them over TCP and HTTP from the internet connection. It's only Kafka which is not working.

Provided that the telnet and nc connections are connecting well, I think this is not a firewall or network routing problem. That leaves me only with the producer code, given below. Is there some type of setting to be applied on Kafka Broker or Kafka Producer side for them to be able to connect over the internet? Same issue with consumer code. It can't connect from internet.

Producer code:

   public static void main(String[] args){

       Properties props = new Properties();
       //props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "DNSHOSTNAME:9092");       
       props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "MYEXTERNALIP:9092");

       props.put(ProducerConfig.ACKS_CONFIG, "all");
       props.put(ProducerConfig.RETRIES_CONFIG, 0);
       props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringSerializer");
       props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringSerializer");

       Producer<String, String> producer = new KafkaProducer<String, String>(props);
       TestCallback callback = new TestCallback();
       Random rnd = new Random();
           
           ProducerRecord<String, String> data = new ProducerRecord<String, String>("TP1", "s2", "TEST DATA" );
           producer.send(data, callback);
       producer.close();
   }


   private static class TestCallback implements Callback {
       @Override
       public void onCompletion(RecordMetadata recordMetadata, Exception e) {
           if (e != null) {
               System.out.println("Error while producing message to topic :" + recordMetadata);
               e.printStackTrace();
           } else {
               String message = String.format("sent message to topic:%s partition:%s  offset:%s", recordMetadata.topic(), recordMetadata.partition(), recordMetadata.offset());
               System.out.println(message);
           }
       }
   }

Troubleshooting steps done:

1. Deployed Kafka on 3 different computers, 2 windows and 1 mac. Same results. Works well from the same network, not when they are on internet. Telnet works with each kafka computer from internet/outside network.
2. Tried 3 different external networks, one being my cell phone's hot spot.
3. Tried other softwares as well on kafka machine, as mentioned above and they work fine.
4. Completely disabled the system firewall just to ensure there is no firewall issue, but no luck.

Can someone please help me here on this thread?

Thanks in advance !

Answer 1

The problem you're describing can be solved by "advertising" an address for each broker over the internet via advertised.listeners

telnet and nc check that port is open (listeners config), but cannot check that the brokers bootstrap correctly back to the client, you can instead use kafkacat -L -b <bootstrap> for that, which will return a list of brokers in the cluster, and connecting to each of those should be possible from where you want to run the client

If you have multiple brokers behind your single WAN address, they'll have to advertise / forward separate ports for full connectivity

---

Alternative solutions include the Kafka REST Proxy

In either case, adding SSL to the connection should be done