How do you obtain underlying failed request data when catching requests.exceptions.RetryError?

Question

I am using a somewhat standard pattern for putting retry behavior around requests requests in Python,

import requests
from requests.adapters import HTTPAdapter
from requests.packages.urllib3.util.retry import Retry

retry_strategy = Retry(
    total=HTTP_RETRY_LIMIT,
    status_forcelist=HTTP_RETRY_CODES,
    method_whitelist=HTTP_RETRY_METHODS,
    backoff_factor=HTTP_BACKOFF_FACTOR
)
adapter = HTTPAdapter(max_retries=retry_strategy)
http = requests.Session()
http.mount("https://", adapter)
http.mount("http://", adapter)

...

try:
    response = http.get(... some request params ...)
except requests.Exceptions.RetryError as err:
    # Do logic with err to perform error handling & logging.

Unfortunately the docs on RetryError don't explain anything and when I intercept the exception object as above, err.response is None. While you can call str(err) to get the message string of the exception, this would require unreasonable string parsing to attempt to recover the specific response details and even if one is willing to try that, the message actually elides the necessary details. For example, one such response from a deliberate call on a site giving 400s (not that you would really retry on this but just for debugging) gives a message of "(Caused by ResponseError('too many 400 error responses'))" - which elides the actual response details, like the requested site's own description text for the nature of the 400 error (which could be critical to determining handling, or even just to pass back for logging the error).

What I want to do is receive the response for the last unsuccessful retry attempt and use the status code and description of that specific failure to determine the handling logic. Even though I want to make it robust behind retries, I still need to know the underlying failure beyond "too many retries" when ultimately handling the error.

Is it possible to extract this information from the exception raised for retries?

Answer 1

We can't get a response in every exception because a request may not have been sent yet or a request or response may not have reached its destination. For example these exceptions dont' get a response.

urllib3.exceptions.ConnectTimeoutError
urllib3.exceptions.SSLError
urllib3.exceptions.NewConnectionError

There's a parameter in urllib3.util.Retry named raise_on_status which defaults to True. If it's made False, urllib3.exceptions.MaxRetryError won't be raised. And if no exceptions are raised it is certain that a response has arrived. It now becomes easy to response.raise_for_status in the else block of the try block wrapped in another try.

I've changed except RetryError to except Exception to catch other exceptions.

import requests
from requests.adapters import HTTPAdapter
from requests.packages.urllib3.util.retry import Retry
from requests.exceptions import RetryError

# DEFAULT_ALLOWED_METHODS = frozenset({'DELETE', 'GET', 'HEAD', 'OPTIONS', 'PUT', 'TRACE'})
#     Default methods to be used for allowed_methods
# RETRY_AFTER_STATUS_CODES = frozenset({413, 429, 503})
#     Default status codes to be used for status_forcelist

HTTP_RETRY_LIMIT = 3
HTTP_BACKOFF_FACTOR = 0.2

retry_strategy = Retry(
    total=HTTP_RETRY_LIMIT,
    backoff_factor=HTTP_BACKOFF_FACTOR,
    raise_on_status=False,
)
adapter = HTTPAdapter(max_retries=retry_strategy)
http = requests.Session()
http.mount("https://", adapter)
http.mount("http://", adapter)
try:
    response = http.get("https://httpbin.org/status/503")
except Exception as err:
    print(err)
else:
    try:
        response.raise_for_status()
    except Exception as e:
        # Do logic with err to perform error handling & logging.
        print(response.reason)
        # Or
        # print(e.response.reason)
    else:
        print(response.text)

Test;

# https://httpbin.org/user-agent
➜  python requests_retry.py
{
  "user-agent": "python-requests/2.28.1"
}

# url =  https://httpbin.org/status/503
➜  python requests_retry.py
SERVICE UNAVAILABLE