does anyone know how docker decides which NIC will work with the docker0 network? I have a node with two interfaces (eth0 and ens34), however, only the requests that go through eth0 are forwarded to the container.
When my VM was provisioned and Docker was installed, I started a very silly test: I created a centos vm, installed netcat on it and committed the image. Then I started a daemon container listening on port 8080. I used:
docker -it -p 8080:8080 --name nc-server nc-server nc -vv -l 8080
So I tried to connect to the container listening on port 8080 from another node in the same network (in the same IP address as the interface ens34). It did not work.
Whereas when I sent a request from another machine to the IP address from eth0, I saw some reaction in the container (the communication worked). I was "tailing" its output with:
docker logs -ft nc-server
My conclusion with this experiment: there's some mysterious relationship between eth0 (primary NIC) and docker0, and the requests that are sent to ens34 (10.) interface are never forwarded to the veth / docker0 interfaces, only the requests that go through eth0 (9.*). Why's that?
Also, I know I can make everything work if I use --net=host, but I don't want to use that... it doesn't feel right somehow, is it a standard practice to use the HOST mode in Docker containers? Any caveats on that?
--
UPDATE: I managed to make it work after disabling iptables:
service iptables stop
However, I still don't get what is going on. The info below should be relevant to understand what is going on:
ifconfig
[root@mydockervm2 myuser]# ifconfig | grep -A 1 flags
docker0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 172.17.0.1 netmask 255.255.0.0 broadcast 0.0.0.0
--
ens34: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.1.21.18 netmask 255.255.255.0 broadcast 10.1.21.255
--
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 9.32.145.99 netmask 255.255.255.0 broadcast 9.32.148.255
--
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
--
veth8dbab2f: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet6 fe80::3815:67ff:fe9b:88e9 prefixlen 64 scopeid 0x20<link>
--
virbr0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
inet 192.168.122.1 netmask 255.255.255.0 broadcast 192.168.122.255
netstat
[root@mydockervm2 myuser]# netstat -nr
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
0.0.0.0 9.32.145.1 0.0.0.0 UG 0 0 0 eth0
9.32.145.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
10.1.21.0 0.0.0.0 255.255.255.0 U 0 0 0 ens34
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ens34
172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0
192.168.122.0 0.0.0.0 255.255.255.0 U 0 0 0 virbr0
filters
[root@mydockervm2 myuser]# iptables -t filter -vS
-P INPUT ACCEPT -c 169 106311
-P FORWARD ACCEPT -c 0 0
-P OUTPUT ACCEPT -c 110 13426
-N DOCKER
-N DOCKER-ISOLATION
-A FORWARD -c 0 0 -j DOCKER-ISOLATION
-A FORWARD -o docker0 -c 0 0 -j DOCKER
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -c 0 0 -j ACCEPT
-A FORWARD -i docker0 ! -o docker0 -c 0 0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -c 0 0 -j ACCEPT
-A FORWARD -m physdev --physdev-is-bridged -c 0 0 -j ACCEPT
-A DOCKER -d 172.17.0.2/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 8080 -c 0 0 -j ACCEPT
-A DOCKER-ISOLATION -c 0 0 -j RETURN
nat
[root@mydockervm2 myuser]# iptables -t nat -vS
-P PREROUTING ACCEPT -c 28 4818
-P INPUT ACCEPT -c 28 4818
-P OUTPUT ACCEPT -c 8 572
-P POSTROUTING ACCEPT -c 8 572
-N DOCKER
-A PREROUTING -m addrtype --dst-type LOCAL -c 2 98 -j DOCKER
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -c 0 0 -j DOCKER
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -c 0 0 -j MASQUERADE
-A POSTROUTING -s 172.17.0.2/32 -d 172.17.0.2/32 -p tcp -m tcp --dport 8080 -c 0 0 -j MASQUERADE
-A DOCKER -i docker0 -c 0 0 -j RETURN
-A DOCKER ! -i docker0 -p tcp -m tcp --dport 8080 -c 0 0 -j DNAT --to-destination 172.17.0.2:8080
Thoughts?
First, rule out the obvious and make sure that hosts on the other networks know how to route to your machine to reach the container network. For that, check
netstat -nr
on the source host and make sure that your docker subnet is listed with your docker host as the gateway, or that the default router handling the traffic upstream knows about your host.
If traffic is getting routed but blocked, then you're getting into forwarding and iptables. For forwarding, the following should show a 1:
cat /proc/sys/net/ipv4/ip_forward
Make sure your local host shows a route for the bridges to your container networks with the same netstat command, there should be a line for the docker0 interface and your docker subnet as the destination:
netstat -nr
For iptables, check to see if there are any interface specific nat or filter rules that need to be adjusted:
iptables -t filter -vS
iptables -t nat -vS
If your forward rule defaults to DROP instead of ACCEPT, you may want to add some logging, or just change the default to accept traffic if you believe it can be trusted (e.g. the host is behind another firewall).
This all being said, advertising ports directly on the host is a fairly common practice with containers. For the private stuff, you can setup multiple containers isolated on their internal network that can talk to each other, but no other containers, and you only expose the ports that are truly open to the rest of the world on the host with the -p flag to the run (or ports option in docker-compose).