I have a function for a scraper that connects to a webpage, and checks the response code (anything within 200 fine, anything else not ok). The function retries the connection, when it has a connection error or an SSLerror, and tries it again and again until the try limit has been reached. Within my try block, I try to validate the response with if else statement. If the response is ok, the response is returned, otherwise, the else statement should print the response code, but also execute the except block. Should this be done by manually raising an exception, and calling this in the except block as such?
#Try to get the url at least ten times, in case it times out
def retries(scraper, json, headers, record_url, tries=10):
for i in range(tries):
try:
response=scraper.post("https://webpageurl/etc", json=json, headers=headers)
if response.ok:
print ('OK!')
return response
else:
print (str(response))
raise Exception("Bad Response Code")
except (Exception, ConnectionError, SSLError):
if i < tries - 1:
sleep(randint(1, 2))
continue
else:
return 'Con error'
Since this is not really an exception, you could just check for a problem condition after the try .. except
block:
#Try to get the url at least ten times, in case it times out
def retries(scraper, json, headers, record_url, tries=10):
problem = False
for i in range(tries):
try:
response=scraper.post("https://webpageurl/etc", json=json, headers=headers)
if response.ok:
print ('OK!')
return response
else:
print (str(response))
problem = True
except (Exception, ConnectionError, SSLError):
problem = True
if problem:
if i < tries - 1:
sleep(randint(1, 2))
continue
else:
return 'Con error'
Since your try..except
is within the for
loop, it would continue regardless, I assumed that was by design.
In the example above, I removed the "Bad Response Code"
text, since you made no use of it anyway (you don't access the exception, and don't reraise it), but of course instead of a flag, you could also pass a specific problem code, or even a message.
The advantage of the if
after the except
is that no exception has to be raised to achieve the same, which is a much more expensive operation.
However, I'm not saying the above is always preferred - the advantage of an exception is that you could reraise it as needed and you can catch it outside the function instead of inside, if the exception would require it.