Moving app to docker from host increases HTTP lag by 5 seconds

Question

I have an app that I noticed was performing badly when making outgoing HTTP requests (an extra lag of 5 seconds). Through a lot of trials / tests, I realized that moving the app outside of docker into the host machine eliminated the weird HTTP lag.

I'm using alpine linux for the docker image, and ubuntu is hosting the parent machine.

Docker Info:

Docker version 1.11.2, build b9f10c9

      "NetworkSettings": {
        "Bridge": "",
        "SandboxID": "3ab81b8a66a99c6e9b1a1f49c5410d8260db37eee96c9231c0d83c1b40f84fa5",
        "HairpinMode": false,
        "LinkLocalIPv6Address": "",
        "LinkLocalIPv6PrefixLen": 0,
        "Ports": {
            "8084/tcp": null
        },
        "SandboxKey": "/var/run/docker/netns/3ab81b8a66a9",
        "SecondaryIPAddresses": null,
        "SecondaryIPv6Addresses": null,

        "EndpointID": "464acfb299941bbd301051ea05451823a7e527161185570c00f8569ce2afde88",
        "Gateway": "172.17.0.1",
        "GlobalIPv6Address": "",
        "GlobalIPv6PrefixLen": 0,
        "IPAddress": "172.17.0.3",
        "IPPrefixLen": 16,
        "IPv6Gateway": "",
        "MacAddress": "02:42:ac:11:00:03",
        "Networks": {
            "bridge": {
                "IPAMConfig": null,
                "Links": null,
                "Aliases": null,
                "NetworkID": "32ebc75bc4c98106c6775905906723405c58bc3de914283234a8e1273cba7193",
                "EndpointID": "464acfb299941bbd301051ea05451823a7e527161185570c00f8569ce2afde88",

                "Gateway": "172.17.0.1",
                "IPAddress": "172.17.0.3",
                "IPPrefixLen": 16,
                "IPv6Gateway": "",
                "GlobalIPv6Address": "",
                "GlobalIPv6PrefixLen": 0,
                "MacAddress": "02:42:ac:11:00:03"
            }
        }
    }

I ran the docker container without anything fancy:

docker run -d test

My code (python) times how long it takes for the whole request cycle:

now = datetime.now()
response = http.request('POST', url, body=request_body, headers=headers)

print(
    "\nTotal Time: ",
    (datetime.now() - now).total_seconds()
)

The constant times are 5-6 seconds, where outside the container, its less than once second.

I did a drill to check DNS and a tcpdump from inside the container. The doesn't seem to be anything interesting to report from there. DNS looks fine, and the packet captures report 0.2 sec requests times.

The only interesting thing I found from the packet capture was that it took 5 seconds to see the beginning of the packet from when the http.request was called.

I'm convinced there is a docker networking misconfiguration here somewhere. Please let me know if there any more information I need to add. We are using docker for 20 other services without this problem.

Thanks!

Answer 1

I don't know whether you still require this information or not. We've seen something similar, which was resolved by upgrading Docker.

Not knowing the cause of the issue caused me to do a little digging, and I found this:

https://github.com/docker/docker/issues/20661

In a nutshell, it has something to do with docker attempting to resolve an ipv6 dns before trying to do the same for ipv4. Have a read. You may understand it better than I did.

I hope it helps you.