Why requests to the forwarded docker port are not blocked despite firewall blocking the port?

Question

I'm trying to set up a simple web application in a docker container with a nginx reverse proxy (not in a container, on the host).

Here is how I run the container:

docker run -d --name mywebapp -v /webapp/:/data/ --restart unless-stopped -p 5665:80 mywebapp/server:latest

The web application listens on port 80 inside the container, that I forward to 5665 host port.

The nginx reverse proxy listens on port 5666 and forwards the requests to the forwarded container port 5665.

The netstat -lntup output (only relevant lines):

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:5666            0.0.0.0:*               LISTEN      29369/nginx: master 
tcp        0      0 0.0.0.0:5665            0.0.0.0:*               LISTEN      151974/docker-proxy 

I want to configure the firewall block all connections from outside except to the 5666 port.

Here are my nftables rules (exact copy, nothing is removed):

define IIFNAME = "ens3";

table inet filter {
  chain input {
    type filter hook input priority filter
    policy drop

    ct state invalid drop comment "early drop of invalid connections"
    ct state {established, related} accept comment "allow tracked connections"
    iifname lo accept comment "allow from loopback"
    ip protocol icmp accept comment "allow icmp"
    meta l4proto ipv6-icmp accept comment "allow icmp v6"
    tcp dport ssh accept comment "allow sshd"
    tcp dport 5666 accept comment "webserver";
    
    pkttype host limit rate 5/second counter reject with icmpx type admin-prohibited
    counter
  }

  chain forward {
    type filter hook forward priority filter
    policy drop

        iifname "docker0" oifname $IIFNAME accept;
        iifname $IIFNAME oifname "docker0" accept;
  }

I can connect to port 5666, which is expected and desired. However, the problem is that despite having connections to 5665 port blocked (at least I thought so), I still can connect to 5665 port, bypassing the nginx reverse proxy.

After some trial and error I found the problem was with the "forward" chain rules, that is:

        iifname "docker0" oifname $IIFNAME accept;
        iifname $IIFNAME oifname "docker0" accept;

Removing these rules fixes this problem, but it also prevents the container from accessing the internet.

My two questions:

1. Why do "forward" rules even matter? I thought that "forward" rules only apply when a packet is supposed to be received by another machine (or a container). However, the 5665 port is listening on the current host port by docker-proxy. Even though the data is "forwarded" to the container, it is not forwarded by the kernel, it is forwarded by a user-space application - docker-proxy, so I'd expect the kernel to use INPUT chain instead of FORWARD chain. Yet, it seems like FORWARD rules do matter.

2. (maybe the answer to this question would be clear having the answer to the previous one, yet I'll ask it) How can I block the access to the forwarded 5665 port from outside (from ens3 interface), while allowing the container to access the internet (that is, make requests through the ens3 interface on the host) ?

Answer 1

Docker uses the firewall to route traffic, so it can get confusing to also use the firewall to restrict access.

An easier solution is just to have docker bind to the localhost interface explicitly with ... -p 127.0.0.1:5665:80. By default, without an address, docker will listen on all interfaces, but you don't want that in this case.