can't access kafka outside the docker container

Question

I installed the latest HDP into the docker container.

docker ps
>hortonworks/sandbox-proxy:1.0
>hortonworks/sandbox-hdp:3.0.1

Kafka broker is running at 6667 port that it's working fine.

   >ssh [email protected] -p 2222
   > echo "send test message" | kafka-console-producer.sh --broker-list sandbox-hdp.hortonworks.com:6667 --topic hotelReservation > /dev/null
   >kafka-console-consumer.sh   --bootstrap-server sandbox-hdp.hortonworks.com:6667 --topic test --from-beginning

The only problem is that I don't have access to the Kafka broker outside the docker container.

More information for troubleshooting:

I verified that all the ports are open local machine -> proxy -> hdp

HDP Server:

      hostname -I
      172.18.0.2

      netstat -vatn | grep 6667
      tcp        0      0 172.18.0.2:6667         0.0.0.0:*               LISTEN

Proxy Server

  hostname -I
  172.18.0.3

  nmap 172.18.0.2 -p 6667
  6667/tcp open  irc

  netstat -vatn | grep 6667
  tcp        0      0 0.0.0.0:6667            0.0.0.0:*               LISTEN

My Local machine:

nmap sandbox-hdp.hortonworks.com -p 6667
    Host is up (0.000064s latency).
    rDNS record for 127.0.0.1: localhost
    PORT     STATE SERVICE
    6667/tcp open  irc

Note: 6668 is open too.

included default stream conf (/etc/nginx/conf/stream.d/tcp-hdp.conf):

server {
  listen 6668;
  proxy_pass sandbox-hdp:6667;
}

So I decided to create new config the http: /etc/nginx/conf/stream.d/tcp-hdp.conf

server {
  listen 6667;
  server_name sandbox-hdp.hortonworks.com;
  location / {
    proxy_pass http://sandbox-hdp:6667;
  }
}

Note: I read (here) that I should use advertised.listeners to access broker outside the doctor:

>listeners=PLAINTEXT://0.0.0.0:6667
>advertised.listeners=PLAINTEXT://172.18.0.2:6667

but I 'm sure that's incorrect for my case. I have access in this network, the problem is some type of Nginx mapping only.

Answer 1

Solution:

Run the following command to find hortonworks/sandbox-proxy container id.

docker ps

Log in to the proxy server

docker exec -it CONTAINER_ID /bin/bash

add new mapping in the nginx

cd /etc/nginx/conf.stream.d
vim tcp-hdp.conf

config:

server {
  listen 6667;
  proxy_pass sandbox-hdp:6667;
}

reload nginx mapping:

/etc/init.d/nginx reload

Test:

> ./kafka-console-consumer.sh --bootstrap-server localhost:6668 --topic topicName --from-beginning
> echo "msg" | ./kafka-console-producer.sh --broker-list sandbox-hdp.hortonworks.com:6668 --topic topicName > /dev/nul

The problem was that 6667:6667 mapping was missing. Now brokers are access both 6667 and 6668 ports.