Search code examples
pythonposttimeouturllib2

Python urllib2 setting timeout


I'm trying to fetch a url by making a POST request using Python's urllib2 module. I'm constructing the request in the following way.

handler = urllib2.HTTPHandler()
opener = urllib2.build_opener(handler)
url = 'xyz...'
request = urllib2.Request(url,data='{}')
request.add_header('Content-Type','application/json')
request.get_method = lambda: 'POST'
try:
   connection = opener.open(request)
except urllib2.HTTPError as e:
   connection = e
except urllib2.URLError as e:
   print 'TIMEOUT: ' + e.reason

I want to set a timeout for the open request someplace. Per the docs https://docs.python.org/3.1/library/urllib.request.html the build_opener() call should return a OpenDirector instance which should have a timeout parameter. But I can't seem to get it to work. Also, the reason I'm constructing a request is because I need to specify an empty body data='{}' in the request and I can't seem to be able to get that going with urlopen either. Any help appreciated.


Solution

  • You can pass timeout as a parameter to the open method call of the opener.

    Normal functioning using lambda function to ensure request is POST rather than GET with no body

    >>> import urllib2
    >>> handler = urllib2.HTTPHandler()
    >>> opener = urllib2.build_opener(handler)
    >>> request = urllib2.Request('http://httpbin.org/post')
    >>> request.get_method = lambda: 'POST'
    >>> opener.open(request)
    <addinfourl at 4363264800 whose fp = <socket._fileobject object at 0x101b654d0>>
    

    Simply add timeout,

    >>> opener.open(request, timeout=0.01)
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 431, in open
        response = self._open(req, data)
      File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 449, in _open
        '_open', req)
      File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 409, in _call_chain
        result = func(*args)
      File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1227, in http_open
        return self.do_open(httplib.HTTPConnection, req)
      File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1197, in do_open
        raise URLError(err)
    urllib2.URLError: <urlopen error timed out>