How to send a repeat request in urllib2.urlopen in Python if the first call is just stuck?

Question

I am making a call to a URL in Python using urllib2.urlopen in a while(True) loop

My URL keeps changing every time (as there is a change in a particular parameter of the URL every time).

My code look as as follows:

def get_url(url):
    '''Get json page data using a specified API url'''
    response = urlopen(url)
    data = str(response.read().decode('utf-8'))
    page = json.loads(data)
    return page

I am calling the above method from the main function by changing the url every time I make the call.

What I observe is that after few calls to the function, suddenly (I don;t know why), the code gets stuck at the statement

response = urlopen(url)

and it just waits and waits...

How do I best handle this situation?

I want to make sure that if it does not respond within say 10 seconds, I make the same call again.

I read about

response = urlopen(url, timeout=10)

but then what about the repeated call if this fails?

Answer 1

Depending on how many retries you want to attempt, use a try/catch inside a loop:

while True:
    try:
        response = urlopen(url, timeout=10)
        break
    except:
        # do something with the error
        pass
# do something with response
data = str(response.read().decode('utf-8'))
...  

This will silence all exceptions, which may not be ideal (more on that here: Handling urllib2's timeout? - Python)