Search code examples
pythonhttptcpkeep-aliveurllib3

Python urllib3: close idle connection after some time

Is there a way to tell Python urllib3 to not reuse idle connections after some period of time, and instead to close them?

Looking in https://urllib3.readthedocs.io/en/latest/reference/index.html#module-urllib3.connectionpool doesn't seem to show anything relevant.

Solution

  • Remember:

    A connection pool is a cache of database connections maintained so that the connections can be "reused" when future requests to the database are required.

    You can do this is many ways (I guess):

    • Set retries to one.

    This breaks your connection if it failed one time. To set it:

    import requests
s = requests.Session()
a = requests.adapters.HTTPAdapter(max_retries=1) # is zero for default
s.mount('http://', a)
    • Change pool connections.

    The "pool_connections" is the number of host-pools to keep around. For example, if you're connecting to 100 different hosts, and pool_connections=10, then only the latest 10 hosts' connections will be re-used. To set that:

    s = requests.Session()
s.mount('https://', HTTPAdapter(pool_connections=1))
s.get('https://www.example.com')

    This will stop the reuse of pools.

    • Pool Maxsize

    This is cared only if you use Session in a multithreaded environment. To set it:

    s = requests.Session()
s.mount('https://', HTTPAdapter(pool_connections=1, pool_maxsize=1))
    • Configure Maxsize

    he :class:~connectionpool.ConnectionPool class keeps a pool of individual :class:~connection.HTTPConnection instances. These connections are used during an individual request and returned to the pool when the request is complete. By default only one connection will be saved for re-use. To set it (it is, by default):

    from urllib3 import HTTPConnectionPool
pool = HTTPConnectionPool('www.example.com', maxsize=0) #likely to slow you down cuz it never stores the pools

    maxsize – Number of connections to save that can be reused. More than 1 is useful in multithreaded situations.

    • Let the Pool Manager do that!

    The PoolManager uses a Least Recently Used (LRU) policy for discarding old pools. That is, if you set the PoolManager num_pools to 10, then after making requests to 11 or more different hosts, the least recently used pools will be cleaned up eventually. So to do that:

    from urllib3 import PoolManager
manager = PoolManager(1) # not the manager cleans up pools used for one time
r = manager.request('GET', 'http://www.example.com/')

    Also, the docs says that:

    Cleanup of stale pools does not happen immediately.

    So for that Use RecentlyUsedContainer (Docs contains only one line).

    Note:

    Setting arguments if PoolManager affects all the pools connected thereby.

    Hopes this helps you. Get the advanced usage docs HERE .