Where does the limitation to 100 connections to docker.socket (using aiodocker) come from?

Question

I'm using the Python-package aiodocker do monitor running Docker containers using container.stats() like this:

async for stats in container.stats():
    do_something(...)

And this works as expected.

lsof +E /run/docker.sock (might also be /var/run/docker.sock) reveals that there will be a connection to /run/docker.sock established for every monitored container - which is not a surprise, docker stats would also use one connection per container.

However, using Python / aiodocker I can establish a maximum of 100 connections per process - after that new calls involving a new connection will block.

Running another instance of the same Python script seems to be not affected by the first one (but can also open a maximum of 100 connections).

Strangely docker stats does not seem to have this limitation - it will happily monitor 200 and more containers.

Searching through the aiodocker source code does not show any hard coded limitation like this, and using DDG/Google/ChatGPT I can't find any reference to a limitation to 100 connections/process.

Where is this limitation defined? Is it configurable?

Answer 1

aiodocker uses aiohttp. aiohttp defaults to a connection limit of 100.

aiohttp creates a client using ClientSession which in turn uses a default BaseConnector.

In the docs, you'll see the limit argument on the BaseConnector says:

limit (int) – total number simultaneous connections. If limit is None the connector has no limit (default: 100).

You can pass in a custom connector. But in this case you also have to define url. On a Linux system the following function returns a Docker() instance with customizable file limit:

def docker_connection(limit: int = 100) -> Docker:
    """Creates a Docker connection with a customized Connector in order to
    increase connection limit"""
    for sockpath in [Path("/run/docker.sock"), Path("/var/run/docker.sock")]:
        if sockpath.is_socket():
            return Docker(
                "unix://localhost",
                aiohttp.UnixConnector(sockpath.as_posix(), limit=limit))
    raise RuntimeError("No path to docker.sock found")

# to be used like this:
async with docker_connection(limit=256) as self.docker_client:
    ...

Documentation on aiodocker client function signature + aiohttp client:

https://aiodocker.readthedocs.io/en/latest/client.html https://docs.aiohttp.org/en/stable/client_reference.html