Docker swarm containers not able to access internet

I am trying to setup a swarm cluster in AWS, however the containers in the host are not able to access the internet. The ping command for both address resolution or direct connectivity via IP is not working from inside the container.

Before creating this ticket I had a look at this issue, but I don't think there is CIDR overlap in my case.

I have the following configurations:

Public Subnet CIDR : 10.2.1.0/24
Namespace server inside this is :10.2.0.2

Ingress overlay network --> 10.255.0.0/16

docker_gwbridge --> 172.18.0.0/1

I have also tried creating the new overlay(192.168.1.0/24) and docker_gwbridge(10.11.0.0/16) network with no luck.

I am creating the service with these options(removing the mount and env parameters):

docker service create --publish 8098:8098 <Imagename>

Please note when I was creating the overlay network by myself I was adding the option --network my-overlay as well in the create command.

Any pointers as to what I might be missing/doing wrong?

Edit 1 Adding more info

Below is the inspect of container when I am not creating a new overlay network and going with the default one:

"NetworkSettings": {
        "Bridge": "",
        "SandboxID": "eb***",
        "HairpinMode": false,
        "LinkLocalIPv6Address": "",
        "LinkLocalIPv6PrefixLen": 0,
        "Ports": {
            "5005/tcp": null,
            "8080/tcp": null
        },
        "SandboxKey": "/var/run/docker/netns/e***9",
        "SecondaryIPAddresses": null,
        "SecondaryIPv6Addresses": null,
        "EndpointID": "",
        "Gateway": "",
        "GlobalIPv6Address": "",
        "GlobalIPv6PrefixLen": 0,
        "IPAddress": "",
        "IPPrefixLen": 0,
        "IPv6Gateway": "",
        "MacAddress": "",
        "Networks": {
            "ingress": {
                "IPAMConfig": {
                    "IPv4Address": "10.255.0.4"
                },
                "Links": null,
                "Aliases": [
                    "30**"
                ],
                "NetworkID": "g7w**",
                "EndpointID": "291***",
                "Gateway": "",
                "IPAddress": "10.255.0.4",
                "IPPrefixLen": 16,
                "IPv6Gateway": "",
                "GlobalIPv6Address": "",
                "GlobalIPv6PrefixLen": 0,
                "MacAddress": "02:4***"
            }

And below is from when I am creating the overlay network:

"Networks": {
            "ingress": {
                "IPAMConfig": {
                    "IPv4Address": "10.255.0.4"
                },
                "Links": null,
                "Aliases": [
                    "42***"
                ],
                "NetworkID": "jl***3",
                "EndpointID": "792***86c",
                "Gateway": "",
                "IPAddress": "10.255.0.4",
                "IPPrefixLen": 16,
                "IPv6Gateway": "",
                "GlobalIPv6Address": "",
                "GlobalIPv6PrefixLen": 0,
                "MacAddress": "02:4***"
            },
            "my-overlay": {
                "IPAMConfig": {
                    "IPv4Address": "192.168.1.3"
                },
                "Links": null,
                "Aliases": [
                    "42**"
                ],
                "NetworkID": "4q***",
                "EndpointID": "4c***503",
                "Gateway": "",
                "IPAddress": "192.168.1.3",
                "IPPrefixLen": 24,
                "IPv6Gateway": "",
                "GlobalIPv6Address": "",
                "GlobalIPv6PrefixLen": 0,
                "MacAddress": "02:4***"
            }

Solution

I am answering my question as I found out that the reason for this behavior was my custom chef recipe for docker installation. I was setting up iptables=false in the docker config and hence it was not working for any docker container other than those in host network mode.

I got the following advice from Bret(Docker champion in docker community) which helped me to get to the root of the problem. In short it was a issue with something I was doing wrongly, however posting the suggestion below in case you want to troubleshoot such issues in future.

Hey Manish,

Suggestion: get a single container working correctly without swarm or overlays before trying them.

so you should be able to just docker run --rm nginx:alpine ping 8.8.8.8 and get a response.

That verifies that containers on that host have a way to the internet.

Then trying docker run --rm nginx:alpine ping google.com and get a response.

That verifies DNS resolution is working.

*Then you can try creating a single overlay network on one node in a single node swarm:*

*docker swarm init *

*docker network create --driver overlay --attachable mynet *

*docker run --rm --network mynet nginx:alpine ping google.com *

That verifies they have internet and DNS on a overlay network.

If you then add multiple nodes and have issues, then you likely need to ensure all swarm nodes can talk over swarm ports, which you find a link to the firewall port list in The Swarm Section under the Creating a 3-Node Swarm Cluster resources.