Search code examples
dockerfailoverconsuldocker-swarm

How to connect Docker Swarm to multiple consul servers for failover?

I am running docker swarm with consul. I have a consul cluster made with 3 nodes connected to each other for failover. The issue is that I can only connect my swarm workers and masters to a single node and if that node goes down then swarm stops working. So how can I connect swarm workers and masters to all my nodes? The following commands if run from the master will set up my swarm environment connected to a single consul server:

#### REFERENCE
# {{master_i}} is the IP address of the master server
# {{consul_i}} is the IP address of the consul server
# {{worker_i}} is the IP address of a worker server


#### START THE MASTER
docker run --restart=unless-stopped --name=swarm-manager0 -d -p 4000:4000 swarm manage -H :4000 --replication \
--advertise {{master_0}}:4000 \
consul://{{consul_0}}:8500

#### START THE WORKERS REMOTELY FROM THE MASTER
docker -H={{worker_0}}:2375 run -d --restart=unless-stopped --name=swarm-worker0 swarm join \
--advertise={{worker_0}}:2375 \
consul://{{consul_0}}:8500/

docker -H={{worker_1}}:2375 run -d --restart=unless-stopped --name=swarm-worker1 swarm join \
--advertise={{worker_1}}:2375 \
consul://{{consul_0}}:8500/

docker -H={{worker_2}}:2375 run -d --restart=unless-stopped --name=swarm-worker2 swarm join \
--advertise={{worker_2}}:2375 \
consul://{{consul_0}}:8500/

#### START THE WORKER SERVICE DISCOVERY
docker -H={{worker_0}}:2375 run -d --restart=unless-stopped \
-h {{worker_0}} --name registrator0 -v /var/run/docker.sock:/tmp/docker.sock gliderlabs/registrator \
consul://{{consul_0}}:8500

docker -H={{worker_1}}:2375 run -d --restart=unless-stopped \
-h {{worker_1}} --name registrator1 -v /var/run/docker.sock:/tmp/docker.sock gliderlabs/registrator \
consul://{{consul_0}}:8500

docker -H={{worker_2}}:2375 run -d --restart=unless-stopped \
-h {{worker_2}} --name registrator2 -v /var/run/docker.sock:/tmp/docker.sock gliderlabs/registrator \
consul://{{consul_0}}:8500

Note that simply adding two extra consul://{{consul_i}}:8500 (for the other two consul servers) to the end of each docker run command will not connect the containers to the other consul servers.

Solution

  • According to @slugonamission there is no way to connect swarm to the multiple IP addresses of multiple consul servers.

    However I was able to create an haproxy load balancer that sat in front of my consul servers. So my load balancer forwarded all traffic from my load balancers port 8500 on to port 8500 on all of my consul servers. By doing this I was able to use the IP address of my load balancer in place of {{CONSUL0}}. Heres my pretty basic haproxy.cfg

    # $CONSUL0 $CONSUL0 and $CONSUL0 are the IP addresses of my consul servers

global
    log 127.0.0.1 local0 notice
    maxconn 2000
    user haproxy
    group haproxy

defaults
    log     global
    mode    http
    option  httplog
    option  dontlognull
    retries 3
    option redispatch
    timeout connect  5000
    timeout client  10000
    timeout server  10000

listen appname 0.0.0.0:8500
    mode http
    stats enable
    stats uri /haproxy?stats
    stats realm Strictly\ Private
    stats auth ubuntu
    balance roundrobin
    option httpclose
    option forwardfor
    server consul0 $CONSUL0:8500 check
    server consul1 $CONSUL1:8500 check
    server consul2 $CONSUL2:8500 check

    After making the change my consul servers can individually go down and swarm will continue working.