As a learning exercise, I'm trying to set up a docker swarm on two test AWS EC2 instances, but I'm running into a problem when I try to access the service from the IP address of the worker node.
On the master server, I ran docker swarm init. Then I took the output token and ran docker swarm join --token <token> <Master Private IP>:2377
Then I did a simple docker service create -p 80:80 --name nginx nginx on the master, followed by a docker service scale nginx=2. Now, checking with docker service ps nginx gives the following:
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
idux5dftj9oj nginx.1 nginx:latest ip-172-31-13-2 Running Running 12 minutes ago
2nwfw3fncybj nginx.2 nginx:latest ip-172-31-14-130 Running Running 38 seconds ago
I've opened the inbound ports on the security groups according to this guide, specifically:
The master and worker servers have the same security group, so I just set the source to itself.
When I run curl http://localhost on the master, it gives me this, which proves it works:
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
<!-- Omitting this for brevity -->
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<!-- Omitting this for brevity -->
</body>
But on the worker, I just get curl: (7) Failed to connect to localhost port 80: Connection refused
A docker ps on the worker gives me:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
b37770b153db nginx:latest "nginx -g 'daemon of…" 34 minutes ago Up 34 minutes 80/tcp nginx.2.2nwfw3fncybjj7qzeierlx0xr
Running docker service inspect nginx on the master gives:
[
{
"ID": "887xm47oavn367w0o4bo1nmce",
"Version": {
"Index": 652
},
"CreatedAt": "2019-05-19T07:50:54.491113206Z",
"UpdatedAt": "2019-05-19T08:02:53.454804111Z",
"Spec": {
"Name": "nginx",
"Labels": {},
"TaskTemplate": {
"ContainerSpec": {
"Image": "nginx:latest@sha256:23b4dcdf0d34d4a129755fc6f52e1c6e23bb34ea011b315d87e193033bcd1b68",
"Init": false,
"StopGracePeriod": 10000000000,
"DNSConfig": {},
"Isolation": "default"
},
"Resources": {
"Limits": {},
"Reservations": {}
},
"RestartPolicy": {
"Condition": "any",
"Delay": 5000000000,
"MaxAttempts": 0
},
"Placement": {
"Platforms": [
{
"Architecture": "amd64",
"OS": "linux"
},
{
"OS": "linux"
},
{
"Architecture": "arm64",
"OS": "linux"
},
{
"Architecture": "386",
"OS": "linux"
},
{
"Architecture": "ppc64le",
"OS": "linux"
},
{
"Architecture": "s390x",
"OS": "linux"
}
]
},
"ForceUpdate": 0,
"Runtime": "container"
},
"Mode": {
"Replicated": {
"Replicas": 2
}
},
"UpdateConfig": {
"Parallelism": 1,
"FailureAction": "pause",
"Monitor": 5000000000,
"MaxFailureRatio": 0,
"Order": "stop-first"
},
"RollbackConfig": {
"Parallelism": 1,
"FailureAction": "pause",
"Monitor": 5000000000,
"MaxFailureRatio": 0,
"Order": "stop-first"
},
"EndpointSpec": {
"Mode": "vip",
"Ports": [
{
"Protocol": "tcp",
"TargetPort": 80,
"PublishedPort": 80,
"PublishMode": "ingress"
}
]
}
},
"PreviousSpec": {
"Name": "nginx",
"Labels": {},
"TaskTemplate": {
"ContainerSpec": {
"Image": "nginx:latest@sha256:23b4dcdf0d34d4a129755fc6f52e1c6e23bb34ea011b315d87e193033bcd1b68",
"Init": false,
"DNSConfig": {},
"Isolation": "default"
},
"Resources": {
"Limits": {},
"Reservations": {}
},
"Placement": {
"Platforms": [
{
"Architecture": "amd64",
"OS": "linux"
},
{
"OS": "linux"
},
{
"Architecture": "arm64",
"OS": "linux"
},
{
"Architecture": "386",
"OS": "linux"
},
{
"Architecture": "ppc64le",
"OS": "linux"
},
{
"Architecture": "s390x",
"OS": "linux"
}
]
},
"ForceUpdate": 0,
"Runtime": "container"
},
"Mode": {
"Replicated": {
"Replicas": 1
}
},
"EndpointSpec": {
"Mode": "vip",
"Ports": [
{
"Protocol": "tcp",
"TargetPort": 80,
"PublishedPort": 80,
"PublishMode": "ingress"
}
]
}
},
"Endpoint": {
"Spec": {
"Mode": "vip",
"Ports": [
{
"Protocol": "tcp",
"TargetPort": 80,
"PublishedPort": 80,
"PublishMode": "ingress"
}
]
},
"Ports": [
{
"Protocol": "tcp",
"TargetPort": 80,
"PublishedPort": 80,
"PublishMode": "ingress"
}
],
"VirtualIPs": [
{
"NetworkID": "6scdvoeno2tviu4zgyldmq6b4",
"Addr": "10.255.0.82/16"
}
]
}
}
]
Here's the master's docker info
Containers: 3
Running: 3
Paused: 0
Stopped: 0
Images: 4
Server Version: 18.09.6
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: active
NodeID: q4h5ahgxf1xwuyi2aotyt20iy
Is Manager: true
ClusterID: r88oqh59x74bl1kqrcg5od2qd
Managers: 1
Nodes: 2
Default Address Pool: 10.0.0.0/8
SubnetSize: 24
Orchestration:
Task History Retention Limit: 5
Raft:
Snapshot Interval: 10000
Number of Old Snapshots to Retain: 0
Heartbeat Tick: 1
Election Tick: 10
Dispatcher:
Heartbeat Period: 5 seconds
CA Configuration:
Expiry Duration: 3 months
Force Rotate: 0
Autolock Managers: false
Root Rotation In Progress: false
Node Address: 172.31.13.2
Manager Addresses:
172.31.13.2:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: bb71b10fd8f58240ca47fbb579b9d1028eea7c84
runc version: 2b18fe1d885ee5083ef9f0838fee39b62d653e30
init version: fec3683
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.15.0-1021-aws
Operating System: Ubuntu 18.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 1.945GiB
Name: ip-172-31-13-2
ID: RM34:I2IM:EJ2V:W74X:ECSD:ABCC:ZB4T:B7UO:OIWW:SUQ2:ILDB:HQLQ
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine
And here's the worker's docker info
Containers: 3
Running: 3
Paused: 0
Stopped: 0
Images: 4
Server Version: 18.09.5
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: active
NodeID: slya32xwjmklumhm23bt7xs6m
Is Manager: false
Node Address: 172.31.14.130
Manager Addresses:
172.31.13.2:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: bb71b10fd8f58240ca47fbb579b9d1028eea7c84
runc version: 2b18fe1d885ee5083ef9f0838fee39b62d653e30
init version: fec3683
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.15.0-1021-aws
Operating System: Ubuntu 18.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 1.945GiB
Name: ip-172-31-14-130
ID: X7FI:3VCW:OCVI:5XSX:HJ24:2NOD:NQYU:SEYL:JVIJ:J4DI:F5UL:NKZT
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Username: bizmd
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine
As far as I've read, there should not be any problems after adding the worker to the swarm and creating a service. Despite that, the worker cannot access the nginx service that it is already hosting.
What could be causing this issue?
I had the idea to check which ports were actually opened in my worker server (as opposed to just which were opened on the firewall).
netstat -tulpn showed me:
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN -
tcp6 0 0 :::9443 :::* LISTEN -
tcp6 0 0 :::22 :::* LISTEN -
udp 19968 0 127.0.0.53:53 0.0.0.0:* -
udp 0 0 172.31.14.130:68 0.0.0.0:* -
udp 0 0 0.0.0.0:4789 0.0.0.0:* -
I noticed that no process was consuming 7946, which is one of the ports that needed to be opened up. So I restarted the docker service: sudo service docker restart
After the restart finished, I saw a process start up and consumed the port. Sure enough, I was then able to execute curl localhost against either node.