I'm trying to test overlay driver concept in multihost docker networking and getting error while registering dochost2 with consul store
dochost1:
root@dochost1:/usr/lib/systemd# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 08:00:27:38:6a:0b brd ff:ff:ff:ff:ff:ff
inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic noprefixroute enp0s3
valid_lft 68198sec preferred_lft 68198sec
inet6 fe80::a00:27ff:fe38:6a0b/64 scope link
valid_lft forever preferred_lft forever
3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 08:00:27:c7:bc:23 brd ff:ff:ff:ff:ff:ff
inet 192.168.56.9/24 brd 192.168.56.255 scope global noprefixroute enp0s8
valid_lft forever preferred_lft forever
inet6 fe80::a00:27ff:fec7:bc23/64 scope link
valid_lft forever preferred_lft forever
4: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 02:42:3c:4a:00:d9 brd ff:ff:ff:ff:ff:ff
inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
valid_lft forever preferred_lft forever
inet6 fe80::42:3cff:fe4a:d9/64 scope link
valid_lft forever preferred_lft forever
24: vetha7949d3@if23: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP group default
link/ether 4e:48:66:9e:d7:14 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet6 fe80::4c48:66ff:fe9e:d714/64 scope link
valid_lft forever preferred_lft forever
dochost2:
root@dochost2:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 08:00:27:a0:17:7a brd ff:ff:ff:ff:ff:ff
inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic noprefixroute enp0s3
valid_lft 82941sec preferred_lft 82941sec
inet6 fe80::c88:231a:9eb7:7d7c/64 scope link noprefixroute
valid_lft forever preferred_lft forever
3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 08:00:27:b8:91:45 brd ff:ff:ff:ff:ff:ff
inet 192.168.50.0/24 brd 192.168.50.255 scope global noprefixroute enp0s8
valid_lft forever preferred_lft forever
inet6 fe80::a00:27ff:feb8:9145/64 scope link
valid_lft forever preferred_lft forever
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:49:2b:5e:ef brd ff:ff:ff:ff:ff:ff
inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
valid_lft forever preferred_lft forever
dochost1 - having consul store and details as below
kernel version:
root@dochost1:/usr/lib/systemd# uname -r
5.0.0-23-generic
ubuntu version:
root@dochost1:/usr/lib/systemd# cat /etc/*release*
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.3 LTS"
NAME="Ubuntu"
VERSION="18.04.3 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.3 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic
docker version:
root@dochost1:/usr/lib/systemd# docker --version
Docker version 19.03.5, build 633a0ea838
root@dochost1:/usr/lib/systemd#
consul container info:
root@dochost1:/usr/lib/systemd# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
237b81df3720 progrium/consul "/bin/start -server …" 2 hours ago Up 2 hours 53/tcp, 53/udp, 8300-8302/tcp, 8400/tcp, 8301-8302/udp, 0.0.0.0:8500->8500/tcp consul
root@dochost1:/usr/lib/systemd#
docker information on dochost1:
root@dochost1:/usr/lib/systemd# docker info
Client:
Debug Mode: false
Server:
Containers: 4
Running: 1
Paused: 0
Stopped: 3
Images: 7
Server Version: 19.03.5
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: b34a5c8af56e510852c35414db4c1f4fa6172339
runc version: 3e425f80a8c931f88e6d94a8c831b9d5aa481657
init version: fec3683
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 5.0.0-23-generic
Operating System: Ubuntu 18.04.3 LTS
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 1.944GiB
Name: dochost1
ID: OCT2:CMAB:WPLU:VDL6:MZNH:CWXM:XMRU:CCHH:NK6S:XYRZ:RCWF:52PQ
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Cluster Store: consul://192.168.56.9:8500/network
Cluster Advertise: 192.168.56.9:0
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No swap limit support
in second docker host node - dochost2, i'm trying to edit /lib/systemd/system/docker.service file execstart as below and restart the docker
ExecStart=/usr/bin/dockerd -H fd:// --cluster-store=consul://192.168.56.9:8500/network --cluster-advertise=enp0s8:2376 --containerd=/run/containerd/containerd.sock
i'm getting the below error in journalctl -u docker output
Dec 17 13:34:41 dochost2 dockerd[2370]: time="2019-12-17T13:34:41.713159844+05:30" level=error msg="discovery error: Get http://192.168.56.9:8500/v1/kv/network/docker/nodes?consistent=: dial tcp 192.168.56.9:8500: i/o timeout"
Dec 17 13:35:11 dochost2 dockerd[2370]: time="2019-12-17T13:35:11.714004242+05:30" level=error msg="discovery error: Put http://192.168.56.9:8500/v1/kv/network/docker/nodes?flags=3304740253564472344: dial tcp 192.168.56.9:8500: i/o timeout"
Dec 17 13:35:41 dochost2 dockerd[2370]: time="2019-12-17T13:35:41.714024951+05:30" level=error msg="discovery error: Unexpected watch error"
There is no firewall configured on both hosts as well
root@dochost1:/usr/lib/systemd# ufw status
Status: inactive
root@dochost1:/usr/lib/systemd#
root@dochost2:~# ufw status
Status: inactive
root@dochost2:~#
It is purely network related issue.Since this 2 docker hosts are in different subnets,there is no network connectivity in between these nodes.So i have used linux bridge concept and added below bridge to make a connectivity in between this 2 docker hosts
on dochost 2:
First i have changed the enp0s8 interface ip to 192.168.50.9,since i can't use 192.168.50.0/24 as interface ip because it is pointing to entire subnet(192.168.50.0 - 255) ip's and we need to assign an ip to bridge
root@dochost2:~# ip addr show enp0s8
3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master host1_bridge1 state UP group default qlen 1000
link/ether 08:00:27:b8:91:45 brd ff:ff:ff:ff:ff:ff
inet 192.168.50.9/24 brd 192.168.50.255 scope global noprefixroute enp0s8
valid_lft forever preferred_lft foreverdebug2: channel 0: window 999381 sent adjust 49195
root@dochost2:~#
ip link add host1_bridge1 type bridge
ip address add 192.168.56.8/24 dev host1_bridge1
ip link set dev enp0s8 master host1_bridge1
ip link set host1_bridge1 up
After completed above steps,i was able to ping and ssh dochost 1 - 192.168.56.9
on dochost1:
Added below route which point to bridge interface ip on dochost2
route add -host 192.168.50.9 gw 192.168.56.8
After the above route,i'm able to connect dochost2 from dochost1
After completed above steps i just reloaded docker daemon on dochost2
systemctl daemon-reload
Checked the logs using journalctl -u docker on dochost2 and not seeing any error now
4346354+05:30" level=info msg="2019/12/19 14:22:59 [INFO] serf: EventMemberJoin: dochost2 192.168.50.9\n"
8140350+05:30" level=info msg="2019/12/19 14:22:59 [INFO] serf: EventMemberJoin: dochost1 192.168.56.9\n"
Even for testing this i have created an overlay network myoverlay on dochost1 and the same i was able to see on dochost2
root@dochost1:~# docker network ls
NETWORK ID NAME DRIVER SCOPE
28e8ae4d1273 bridge bridge local
b5b6ab1e41d2 host host local
e49864108832 myoverlay overlay global
8c156d319aa8 none null local
root@dochost1:~#
root@dochost2:~# docker network ls
NETWORK ID NAME DRIVER SCOPE
e3c773160654 bridge bridge local
8bb2cb8d6e5c host host local
e49864108832 myoverlay overlay global
3427614365c3 none null local
root@dochost2:~#