Search code examples
pythonif-statementweb-scrapinghttpresponsetry-except

How to use (if) else in try block to route to except block if else statement is true in pythonic fashion?


I have a function for a scraper that connects to a webpage, and checks the response code (anything within 200 fine, anything else not ok). The function retries the connection, when it has a connection error or an SSLerror, and tries it again and again until the try limit has been reached. Within my try block, I try to validate the response with if else statement. If the response is ok, the response is returned, otherwise, the else statement should print the response code, but also execute the except block. Should this be done by manually raising an exception, and calling this in the except block as such?

#Try to get the url at least ten times, in case it times out
def retries(scraper, json, headers, record_url, tries=10):
    for i in range(tries):
        try:
            response=scraper.post("https://webpageurl/etc", json=json, headers=headers)
            if response.ok:
                print ('OK!')
                return response
            else:
                print (str(response))
                raise Exception("Bad Response Code")

        except (Exception, ConnectionError, SSLError):
            if i < tries - 1:
                sleep(randint(1, 2))
                continue
            else:
                return 'Con error'

Solution

  • Since this is not really an exception, you could just check for a problem condition after the try .. except block:

    #Try to get the url at least ten times, in case it times out
    def retries(scraper, json, headers, record_url, tries=10):
        problem = False
        for i in range(tries):
            try:
                response=scraper.post("https://webpageurl/etc", json=json, headers=headers)
                if response.ok:
                    print ('OK!')
                    return response
                else:
                    print (str(response))
                    problem = True
    
            except (Exception, ConnectionError, SSLError):
                problem = True
    
            if problem:
                if i < tries - 1:
                    sleep(randint(1, 2))
                    continue
                else:
                    return 'Con error'
    

    Since your try..except is within the for loop, it would continue regardless, I assumed that was by design.

    In the example above, I removed the "Bad Response Code" text, since you made no use of it anyway (you don't access the exception, and don't reraise it), but of course instead of a flag, you could also pass a specific problem code, or even a message.

    The advantage of the if after the except is that no exception has to be raised to achieve the same, which is a much more expensive operation.

    However, I'm not saying the above is always preferred - the advantage of an exception is that you could reraise it as needed and you can catch it outside the function instead of inside, if the exception would require it.