Search code examples
pythonwhile-loopempty-list

While Not Loop for empty list in python


I am making a request to a server... for whatever reason (beyond my comprehension), the server will give me a status code of 200, but when I use Beautiful Soup to grab a list from the html, nothing is returned. It only happens on the first page of pagination.

To get around a known bug, I have to loop until the list is not empty.

This works, but it's clunky. Is there a better way to do this? Knowing that I have to force the request until the list contains an item.

# look for attractions
attraction_list = soup.find_all(attrs={'class': 'listing_title'})
while not attraction_list:
    print('the list is empty')
    try:
        t = requests.Session()
        t.cookies.set_policy(BlockAll)
        page2 = t.get(search_url)
        print(page2.status_code)
        soup2 = BeautifulSoup(page2.content, 'html.parser')
        attraction_list = soup2.find_all(attrs={'class': 'listing_title'}) 
    except:
        pass

Solution

  • I came up with this.

    attraction_list = soup.find_all(attrs={'class': 'listing_title'})
    while not attraction_list:
        print('the list is empty')
        for q in range(0, 4):
            try:
                t = requests.Session()
                t.cookies.set_policy(BlockAll)
                page2 = t.get(search_url)
                print(page2.status_code)
                soup2 = BeautifulSoup(page2.content, 'html.parser')
                attraction_list = soup2.find_all(attrs={'class': 'listing_title'})
            except Exception as str_error:
                print('FAILED TO FIND ATTRACTIONS')
                time.sleep(3)
                continue
            else:
                break
    

    It'll try 4 times to get that pull the attractions, if attractions_list ends up with a valid list, it breaks. Good enough.