I am starting a container with kafka via docker-compose successfully. By successfully I mean, I can log in such kafka container and produce and consume messages straight from there. Also I can see another container (filebeat) depending on such kafka container connecting successfully.
docker-compose.yml
version: '3.2'
services:
zoo1:
image: elevy/zookeeper:latest
environment:
MYID: 1
SERVERS: zoo1
ports:
- "2181:2181"
kafka1:
image: wurstmeister/kafka
command: [start-kafka.sh]
depends_on:
- zoo1
links:
- zoo1
ports:
- "9092:9092"
environment:
KAFKA_LISTENERS: PLAINTEXT://:9092
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka1:9092
KAFKA_BROKER_ID: 1
KAFKA_ADVERTISED_PORT: 9092
KAFKA_LOG_RETENTION_HOURS: "168"
KAFKA_LOG_RETENTION_BYTES: "100000000"
KAFKA_ZOOKEEPER_CONNECT: zoo1:2181
KAFKA_CREATE_TOPICS: "log:1:1"
KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'true'
filebeat:
image: docker.elastic.co/beats/filebeat:7.5.2
command: filebeat -e -strict.perms=false
volumes:
- "//c/Users/Cast/megalog-try-1/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro"
- "//c/Users/Cast/megalog-try-1/sample-logs:/sample-logs"
links:
- kafka1
depends_on:
- kafka1
Nevertheless, when I try produce a very simple message from my local kafka to this container kafka I get connection refused.
I found someone with same exception but a bit similar scenario. I tried his solution without success. Basically it points to "adjust advertised.listeners to be the external IP". I did it in my docker-compose but nothing changed at all. And even if this had worked, I guess it would not be a final solution because for me since the guy is hardcoding his computer ip address in KAFKA_ADVERTISED_LISTENERS
So my question is how can I connect from my local kafka running in my Windows 10 to a kafka container in order to send a message?
Here are everything I have tried so far:
1 - trying with local ip
ipconfig
...
Endereço IPv4. . . . . . . . . . . . . . . : 192.168.129.97
kafka-console-producer.bat --broker-list 192.168.129.97:9092 --topic log
C:\tools\kafka\bin\windows>kafka-console-producer.bat --broker-list 192.168.129.97:9092 --topic log
>[2020-03-02 16:54:44,155] WARN [Producer clientId=console-producer] Connection to node -1 (/192.168.129.97:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2020-03-02 16:54:46,212] WARN [Producer clientId=console-producer] Connection to node -1 (/192.168.129.97:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
2 - trying with docker machine ip
C:\tools\kafka\bin\windows>docker-machine ip
192.168.99.100
kafka-console-producer.bat --broker-list 192.168.99.100:9092 --topic log
[2020-03-02 17:00:22,500] WARN [Producer clientId=console-producer] Error connecting to node kafka1:9092 (id: 1 rack: null) (org.apache.kafka.clients.NetworkClient)
java.net.UnknownHostException: kafka1
at java.base/java.net.InetAddress$CachedAddresses.get(InetAddress.java:797)
at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1505)
at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1364)
at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1298)
at org.apache.kafka.clients.ClientUtils.resolve(ClientUtils.java:104)
at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.currentAddress(ClusterConnectionStates.java:403)
at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.access$200(ClusterConnectionStates.java:363)
at org.apache.kafka.clients.ClusterConnectionStates.currentAddress(ClusterConnectionStates.java:151)
at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:955)
at org.apache.kafka.clients.NetworkClient.access$600(NetworkClient.java:69)
at org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:1126)
at org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:1017)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:538)
at org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:335)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:244)
at java.base/java.lang.Thread.run(Thread.java:834)
3 - trying with container ip
docker ps
...
3eb114e24b53 wurstmeister/kafka "start-kafka.sh" 0.0.0.0:9092->9092/tcp megalog-try-1_kafka1_1
docker inspect 3eb114e24b53
...
"IPAddress": "172.18.0.4",
C:\tools\kafka\bin\windows>kafka-console-producer.bat --broker-list 172.18.0.4:9092 --topic log
>testing
[2020-03-02 16:57:54,471] WARN [Producer clientId=console-producer] Connection to node -1 (/172.18.0.4:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
org.apache.kafka.common.KafkaException: Producer closed while send in progress
at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:888)
*** added after CRU's suggestion
C:\tools\kafka\bin\windows>docker exec -it megalog-try-1_zoo1_1 bash
bash-4.3# ls
bin entrypoint.sh home lib64 mnt root sbin sys usr zookeeper
dev etc lib media proc run srv tmp var
bash-4.3# cd zookeeper/
bash-4.3# ls
bin conf data lib log wal zookeeper-3.4.10.jar
bash-4.3# cd conf
bash-4.3# ls
log4j.properties zoo.cfg
bash-4.3# cat zoo.cfg
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/zookeeper/data
dataLogDir=/zookeeper/wal
#snapCount=100000
autopurge.purgeInterval=1
clientPort=2181
quorumListenOnAllIPs=true
server.1=zoo1:2888:3888bash-4.3#
and from kafka container
C:\tools\kafka\bin\windows>docker exec -it megalog-try-1_kafka1_1 bash
bash-4.4# pwd
/
bash-4.4# ls
bin dev etc home kafka lib lib64 media mnt opt proc root run sbin srv sys tmp usr var
bash-4.4# cd kafka
bash-4.4# ls
kafka-logs-0f385d1d435e
bash-4.4# cd kafka-logs-0f385d1d435e/
bash-4.4# ls
app_logs-0 log-0 meta.properties replication-offset-checkpoint
cleaner-offset-checkpoint log-start-offset-checkpoint recovery-point-offset-checkpoint request_logs-0
bash-4.4# cat meta.properties
#
#Mon Mar 02 21:09:33 GMT 2020
cluster.id=-qitBVmjSUGo7Zd7P5cetw
version=0
broker.id=1
bash-4.4#
*** added
After stumbled in this article I could write a docker-compose to run kafka container that my local SpringBoot can send message. Nevertheless, I have to hard code the docker-machine ip manually in my docker-compose. Well, I don't think this is the only way otherwise everytime I start Docker ToolBox I will have to run docker-machine ip and replace the IP Address. I am wondering if there is a more friendly solution.
Bellow 192.168.99.100 is my docker-machine ip.
version: "3.2"
services:
zookeeper:
image: wurstmeister/zookeeper
ports:
- "2181:2181"
kafka:
image: wurstmeister/kafka
ports:
- "9092:9092"
environment:
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_LISTENERS:
"INTERNAL://kafka:9090,\
EXTERNAL://:9092"
KAFKA_ADVERTISED_LISTENERS:
"INTERNAL://kafka:9090,\
EXTERNAL://192.168.99.100:9092"
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP:
"INTERNAL:PLAINTEXT,\
EXTERNAL:PLAINTEXT"
KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
Based on Cricket_007's suggestion, I read carefully another article but it drove me the conclusion that I must use DNS resolution but I can't setup an exclusive dns resolution in my Windows machine as far as I understand.
KAFKA_ADVERTISED_PORT is deprecated. Just use the advertised listeners.
Your advertised listeners are the Docker service name, not a resolvable address to any code running outside of a container on that same Docker network.
You'll want to add in KAFKA_LISTENER_SECURITY_PROTOCOL_MAP
Just use an existing, functional Docker Compose
Btw, links is a deprecated Compose option too