I have two servers running Ubuntu 20.04 (fresh intallation), each has Docker installed. One of the servers (Server A) hosts Rancher 1.6 (rancher/server), while another one (Server B) has rancher/agent. Both servers have an Nginx container installed in Rancher, and the Nginx containers on both servers are configured with port forwarding set to 80->80 and 443->443.
Problem:
Server A: Everything works as expected without any issues.
Server B: It doesn't respond to requests on port 80 or any other port.
I have tried various methods to make it work. If I manually start a container without Rancher, using a command like docker run -d -p 8081:80 --name my-service nginx:latest, it handles requests on port 8081 and displays Ngnix welcome page.
However, if I change the command to docker run -d -p 80:80 --name my-service nginx:latest — I am unable to reach the host.
When using tcpdump port 80, I can see that the requests are reaching the server.
The command netstat -ltnp | grep -w ':80' gives the following output:
tcp6 0 0 :::80 :::* LISTEN 10038/docker-proxy
Going back to Rancher: If I change container's network settings from "Managed" to "Bridge", it successfully forwards any port except for port 80. For example: 8082->80 works fine, 8000->80 works fine, but 80->80 results in "Host unreachable" error. If I change network settings back to "Managed", no port forwarding works at all. I don't understand why this issue is occurring because on Server A, everything works fine with the default settings.
I have spent a couple of days already reading about iptables and other related topics, reinstalling Ubuntu, Docker, Rancher, but I have not achieved any results.
I hope someone can help me identify the problem.
UPD: after reinstalling Ubuntu and installing Docker and rancher/agent I found out that the 80th port got blocked after ipsec-* containers (created by rancher/agent) started. It haven't helped to solve the problem yet, however the breakpoint has been found.
Here are two outputs of iptables -L: before rancher/agent installed — https://pastebin.com/UtYjnqQM and after — https://pastebin.com/tpatsQfU
Well, since I haven't found an answer to this question, I'm posting the solution I found.
As I mentioned before, Server B stopped responding to requests on port 80 after rancher/agent had started its services. When I ran the command iptables -nvL -t nat | grep docker0 I received a response that had two peculiar rows:
0 0 DNAT tcp -- !docker0 * 0.0.0.0/0 0.0.0.0/0 tcp dpt:80 to:10.42.125.250:80
0 0 DNAT tcp -- !docker0 * 0.0.0.0/0 0.0.0.0/0 tcp dpt:443 to:10.42.125.250:443
These entries seemed strange to me because I hadn't started any containers using those ports on the server. However on Server A, where rancher/server is running, there is a Nginx container that uses ports 80 and 443. So, I stopped that container on Server A, and those entries dissapered. Then, I added an Nginx container on Server B with port forwarding set to 80:80 using the Rancher UI, and it became accessible. Finally, I started Nginx container on Server A, and now they work independently, forwarding ports on the respective servers where they reside.
It seems that Rancher propagates iptables rules to other containers if there is no service using ports 80 and 443.
I hope this information is helpful.