Search code examples
pythonurllib3

Seeing retry of a request sent using urllib3.PoolManager without retries configured

I have some python code that looks like the following:

import urllib3
http = urllib3.PoolManager(cert_reqs='CERT_NONE')
...
full_url = 'https://[%s]:%d%s%s' % \
            (address, port, base_uri, relative_uri)
kwargs = {
    'headers': {
        'Host': '%s:%d' % (hostname, port)
    }
}

if data is not None:
    kwargs['body'] = json.dumps(data, indent=2, sort_keys=True)

# Directly use request_encode_url instead of request because requests
# will try to encode the body as 'multipart/form-data'.
response = http.request_encode_url('POST', full_url, **kwargs)
log.debug('Received response: HTTP status %d. Body: %s' %
          (response.status, repr(response.data)))

I have a log line that prints once prior to the code that issues the request, and the log.debug('Received...') line prints once. However, on the server side, I occasionally see two requests (they are both the same POST request that is sent by this code block), around 1-5 seconds apart. In such instances, the order of events is as follows:

  1. One request sent from python client
  2. First request received
  3. Second request received
  4. First response sent with status 200 and an http entity indicating success
  5. Second response sent with status 200 and http entity indicating failure
  6. Python client receives the second reponse

I tried to reproduce it reliably by sleeping in the server (guessing that there might be a timeout that causes a retry), but was unsuccessful. I believe the duplication is unlikely to be occurring on the server because it's just a basic Scala Spray server and haven't seen this with other clients. Looking at the source code for PoolManager, I can't find anywhere where retries would be included. There is a mechanism for retries specified with an optional parameter, but this optional parameter is not being used in the code above.

Does anyone have any ideas where this extra request might be coming from?

EDIT: @shazow gave a pointer about retries having a default of 3, but I changed the code as suggested and got the following error:

Traceback (most recent call last):
  File "my_file.py", line 23, in <module>
    response = http.request_encode_url('GET', full_url, **kwargs)
  File "/usr/lib/python2.7/dist-packages/urllib3/request.py", line 88, in request_encode_url
    return self.urlopen(method, url, **urlopen_kw)
  File "/usr/lib/python2.7/dist-packages/urllib3/poolmanager.py", line 145, in urlopen
    conn = self.connection_from_host(u.host, port=u.port, scheme=u.scheme)
  File "/usr/lib/python2.7/dist-packages/urllib3/poolmanager.py", line 119, in connection_from_host
    pool = self._new_pool(scheme, host, port)
  File "/usr/lib/python2.7/dist-packages/urllib3/poolmanager.py", line 86, in _new_pool
    return pool_cls(host, port, **kwargs)
TypeError: __init__() got an unexpected keyword argument 'retries'`

Edit #2: The following change to kwargs seems to work for me:

import urllib3
http = urllib3.PoolManager(cert_reqs='CERT_NONE')
...
full_url = 'https://[%s]:%d%s%s' % \
            (address, port, base_uri, relative_uri)
kwargs = {
    'headers': {
        'Host': '%s:%d' % (hostname, port)
    },
    'retries': 0
}

if data is not None:
    kwargs['body'] = json.dumps(data, indent=2, sort_keys=True)

# Directly use request_encode_url instead of request because requests
# will try to encode the body as 'multipart/form-data'.
response = http.request_encode_url('POST', full_url, **kwargs)
log.debug('Received response: HTTP status %d. Body: %s' %
          (response.status, repr(response.data)))
Solution

  • urllib3 has a default retries configuration, which is the equivalent to Retry(3). To disable retries outright, you'll need to pass retries=False either when constructing the pool or making a request.

    Something like this should work, for example:

    import urllib3
http = urllib3.PoolManager(cert_reqs='CERT_NONE', retries=False)
...

    The default retries setting (as defined here) could definitely be better documented, I would appreciate your contribution if you feel up for it. :)