Search code examples
dockerdocker-swarm

Docker swarm on 'dind' images and networking problems

I am trying to create a local swarm environment based on 'dind' images. Below are the steps for environment recreation:

docker network create --attachable --subnet 10.0.0.0/16 tools_network

docker run -d --privileged --name swarm-manager-1 --hostname swarm-manager-1 --network tools_network --ip 10.0.0.3 -p 42421:2375 docker:17.03.1-dind

docker --host localhost:42421 swarm init --advertise-addr 10.0.0.3

docker run -d --privileged --name swarm-worker-1 --hostname swarm-worker-1 --network tools_network --ip 10.0.0.4 -p 42423:2375 docker:17.03.1-dind

docker --host localhost:42423 swarm join --token <swarm-token> 10.0.0.3:2377

After that I add an nginx-based proxy:

docker run -d --name swarm-proxy --network tools_network -v $(pwd)/temp-proxy:/etc/nginx:ro -p 80:80 nginx:stable-alpine

with the following contents of nginx.conf in the 'temp-proxy' folder:

events {
    worker_connections  1024;
}

http {
    upstream swarm {
        server 10.0.0.3;
        server 10.0.0.4;

    }

    server {
        listen          80 default_server;
        listen          [::]:80 default_server;

        location / {
            proxy_pass      http://swarm;
            proxy_set_header Host $host;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        }
    }
}

The service I use for testing purposes is launched by: 

docker --host localhost:42421 service create --name test-web --publish 80:80 yeasy/simple-web

What I expect at this stage, based on the mesh networking documentation is that curl localhost will return the result from the deployed service. However, I get a 502 Bad Gateway response, with the following log messages from the proxy: 

[error] 7#7: *4 connect() failed (111: Connection refused) while connecting to upstream, client: 10.0.0.1, server: , request: "GET / HTTP/1.1", upstream: "http://10.0.0.4:80/", host: "localhost"
[error] 7#7: *4 connect() failed (111: Connection refused) while connecting to upstream, client: 10.0.0.1, server: , request: "GET / HTTP/1.1", upstream: "http://10.0.0.3:80/", host: "localhost"
10.0.0.1 - -  "GET / HTTP/1.1" 502 173 "-" "curl/7.47.0"

Same result (connection refused) for curling from a docker container deployed to the tools_network.

Running netstat -l from inside one of the swarm nodes shows that they don't listen on port 80:

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       
tcp        0      0 127.0.0.11:38943        0.0.0.0:*               LISTEN      
tcp        0      0 :::2375                 :::*                    LISTEN      
tcp        0      0 :::2377                 :::*                    LISTEN      
tcp        0      0 :::7946                 :::*                    LISTEN      
udp        0      0 127.0.0.11:35094        0.0.0.0:*                           
udp        0      0 0.0.0.0:4789            0.0.0.0:*                           
udp        0      0 :::7946                 :::*

The questions are:

  1. Is there something wrong with my configuration? What is it?

  2. If not, what steps should I take to get to the problem's root?

Solution

  • After a lot of investigations I found the problem looking at this. Docker inside dind images couldn't initialize properly due to absent modules and I actually saw the errors in the container logs (docker logs swarm-manager-1) but didn't pay attention to them.

    So the solution for me was launching the swarm nodes like this:

    docker run -d --privileged --name swarm-manager-1 --hostname swarm-manager-1 --network tools_network --ip 10.0.0.3 -p 42421:2375 -v /lib/modules:/lib/modules:ro docker:17.03.1-dind

    where the /lib/modules mapping is the piece that was absent.