Cannot setup multi-host Docker overlay network with etcd

I am trying to connect two Docker hosts with an overlay network and am using etcd as a KV-store. etcd is running directly on the first host (not in a container). I finally managed to connect the Docker daemon of the first host to etcd but cannot manage to establish a connection the Docker daemon on the second host.

I downloaded etcd from the Github releases page and followed the instructions under the "Linux" section. After starting etcd, it is listening to the following ports:

etcdmain: listening for peers on http://localhost:2380
etcdmain: listening for peers on http://localhost:7001
etcdmain: listening for client requests on http://localhost:2379
etcdmain: listening for client requests on http://localhost:4001

And I started the Docker daemon on the first host (on which etcd is running as well) like this:

docker daemon --cluster-advertise eth1:2379 --cluster-store etcd://127.0.0.1:2379

After that, I could also create an overlay network with:

docker network create -d overlay <network name>

But I can't figure out how to start the daemon on the second host. No matter which values I tried for --cluster-advertise and --cluster-store, I keep getting the following error message:

discovery error: client: etcd cluster is unavailable or misconfigured

Both my hosts are using the eth1 interface. The IP of host1 is 10.10.10.10 and the IP of host2 is 10.10.10.20. I already ran iperf to make sure they can connect to each other.

Any ideas?

Solution

So I finally figured out how to connect the two hosts and to be honest, I don't understand why it took me so long to solve the problem. But in case other people run into the same problem I will post my solution here. As mentioned earlier, I downloaded etcd from the Github release page and extracted the tar file.

I followed the instructions from the etcd documentation and applied it to my situation. Instead of running etcd with all the options directly from the command line I created a simple bash script. This makes it a lot easier to adjust the options and rerun the command. Once you figured out the right options it would be handy to place them separately in a config file and run etcd as a service as explaind in this tutorial. So here is my bash script:

#!/bin/bash

./etcd --name infra0 \
  --initial-advertise-peer-urls http://10.10.10.10:2380 \
  --listen-peer-urls http://10.10.10.10:2380 \
  --listen-client-urls http://10.10.10.10:2379,http://127.0.0.1:2379 \
  --advertise-client-urls http://10.10.10.10:2379 \
  --initial-cluster-token etcd-cluster-1 \
  --initial-cluster infra0=http://10.10.10.10:2380,infra1=http://10.10.10.20:2380 \
  --initial-cluster-state new

I placed this file in the etcd-vX.X.X-linux-amd64 directory (that I just downloaded and extracted) which also contains the etcd binary. On the second host I did the same thing but changed the --name from infra0 to infra1 and adjusted the IP to that one the second host (10.10.10.20). The --initial-cluster option is not modified.

Then I executed the script on host1 first and then on host2. I'm not sure if the order matters, but in my case I got an error message when I did it the other way round.

To make sure your cluster is set up correctly you can run:

./etcdctl cluster-health

If the output looks similar to this (listing the two members) it should work.

member 357e60d488ae5ab3 is healthy: got healthy result from http://10.10.10.10:2379
member 590f234979b9a5ee is healthy: got healthy result from http://10.10.10.20:2379

If you want to be really sure, add a value to your store on host1 and retrieve it on host2:

host1$ ./etcdctl set myKey myValue
host2$ ./etcdctl get myKey

Setting up docker overlay network

In order to set up a docker overlay network I had to restart the Docker daemon with the --cluster-store and --cluster-advertise options. My solution is probably not the cleanest one but it works. So on both hosts first stopped the docker service and then restarted the daemon with the options:

sudo service docker stop
sudo /usr/bin/docker daemon --cluster-store=etcd://10.10.10.10:2379 --cluster-advertise=10.10.10.10:2379

Note that on host2 the IP addresses need to be adjusted. Then I created the overlay network like this on one of the hosts:

sudo docker network create -d overlay <network name>

If everything worked correctly, the overlay network can now be seen on the other host. Check with this command:

sudo docker network ls