In the docker swarm mode, we are running 2 containers, say container1 & container2, and make it join the same overlay network.
container 1 is not able to resolve the host for container2 and getting time out error if the external dns is not setup.
My question is why external DNS would be required as Embedded DNS of docker should be able to resolve the ip for the other container.
After some debugging and checking the docker daemon logs I figured out when container1 tries to hit container2 , it get the IP correctly for container2 but somehow it reverses the IP and tries to resolve that again which gets timeout of the external DNS is not set and it tries to connect to google DNS.
Following are the logs attached for the docker daemon when container1 tries to reach to container2:-
May 10 08:18:19 machinename dockerd[7426]: time="2019-05-10T08:18:19.762489062-04:00" level=debug msg="Name To resolve: container2."
May 10 08:18:19 machinename dockerd[7426]: time="2019-05-10T08:18:19.762587838-04:00" level=debug msg="[resolver] lookup for container2.: IP [10.0.1.15]"
May 10 08:18:19 machinename dockerd[7426]: time="2019-05-10T08:18:19.799601523-04:00" level=debug msg="IP To resolve 15.1.0.10"
May 10 08:18:19 machinename dockerd[7426]: time="2019-05-10T08:18:19.799930743-04:00" level=debug msg="[resolver] query 15.1.0.10.in-addr.arpa. (PTR) from 172.18.0.8:43634, forwarding to udp:8.8.8.8"
May 10 08:18:23 machinename dockerd[7426]: time="2019-05-10T08:18:23.800300048-04:00" level=debug msg="[resolver] read from DNS server failed, read udp 172.18.0.8:43634->8.8.8.8:53: i/o timeout"
May 10 08:18:23 machinename dockerd[7426]: time="2019-05-10T08:18:23.800693058-04:00" level=debug msg="[resolver] query 15.1.0.10.in-addr.arpa. (PTR) from 172.18.0.8:36077, forwarding to udp:8.8.4.4"
May 10 08:18:27 machinename dockerd[7426]: time="2019-05-10T08:18:27.800992754-04:00" level=debug msg="[resolver] read from DNS server failed, read udp 172.18.0.8:36077->8.8.4.4:53: i/o timeout"
As you can see in the above logs it gets the IP 10.0.1.15 for the container2 but in the very next line it tries to resolve 15.1.0.10, just the reverse of the IP.
Any suggestions to resolve this?
It seems it is resolving through reverse DNS and the ip is showed in the reverse order as you can see in the following post:- https://github.com/moby/moby/issues/20847 Do comment back if you have some other reason for it