I am trying to containerize a java server application which uses kafka. I observed this behavior,
Case 1 : Running as a single container uses bridge network
In this case it works well. So no problem.
Case 2 : Running as a single service uses overlay network (Swarm mode)
In this case, I got org.apache.kafka.common.errors.TimeoutException because of this unsuccessful connection,
WARN [2019-12-06T14:05:44,668] RequestSendThread: warn(): [Controller-0-to-broker-0-send-thread]: Controller 0's connection to broker 10.255.3.64:10101 (id: 0 rack: null) was unsuccessful
java.net.SocketTimeoutException: Failed to connect within 30000 ms
at kafka.controller.RequestSendThread.brokerReady(ControllerChannelManager.scala:270) [kafka_2.12-1.0.0.jar:?]
at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:223) [kafka_2.12-1.0.0.jar:?]
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:64) [kafka_2.12-1.0.0.jar:?]
I don't know much about kafka and can't understand How overlay network affects the kafka process while it works well in bridge network ?
Edit 1:
Almost found the problem... In docker swarm mode, container have two ip address because of the two interfaces bridge and overlay. The ips are endpoint:172.19.0.3, 10.255.3.65.
While trying to connect to the container (inside from the same container),
Bridge I/F : 172.19.0.3 : 10101 - can connect. means telent 172.19.0.3 10101 works.
Overlay I/F :10.155.3.65 : 10101 - can't connect to the port, but can ping works.
The virtual ip 10.155.3.65 which is used by kafka that makes TimeoutException.
Cannot connect to the localhost port while using overlay ip (but can ping). Why this scenario happens ?
Answer:
My bad. I have used default overlay that made the problem.. https://docs.docker.com/network/overlay/ .
Created an --attachable custom overlay network solved this issue.
docker network create -d overlay --attachable my-attachable-overlay