How to handle the error out of a try loop using Selenium and Python

Question

I'd like to run a search with selenium and click the "more results" button at the end of a DDG search.

The DDG search no longer shows the button when it's shown all the results for a query.

I'd like to exit out of the try loop in the case where there is no button.

I'll share what I'm trying now. I also tried earlier these two options: If len(button_element) > 0: button_element.click() and I tried If button_element is not None: button_element.click().

I'd like the solution to use Selenium so it shows the browser because it's helpful for debugging

This is my code with a reproducible example:

    from selenium import webdriver
    from selenium.webdriver.common.keys import Keys
    from selenium.webdriver.chrome.options import Options
    from bs4 import BeautifulSoup

    browser = webdriver.Chrome()        
    browser.get("https://duckduckgo.com/")
    search = browser.find_element_by_name('q')
    search.send_keys("this is a search" + Keys.RETURN)
    html = browser.page_source

    try:
        button_element = browser.find_element_by_class_name('result--more__btn')

        try:
            button_element.click()
        except SystemExit:
            print("No more pages")

    except:
        pass

Answer 1

To click the More Results button at the end of a duckduckgo search results using Selenium WebDriver you have to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following Locator Strategies:

• Code Block:

  from selenium import webdriver
  from selenium.webdriver.support.ui import WebDriverWait
  from selenium.webdriver.common.by import By
  from selenium.webdriver.support import expected_conditions as EC
  from selenium.webdriver.common.keys import Keys
  from selenium.common.exceptions import TimeoutException
  
  options = webdriver.ChromeOptions() 
  options.add_argument("start-maximized")
  options.add_experimental_option("excludeSwitches", ["enable-automation"])
  options.add_experimental_option('useAutomationExtension', False)
  driver = webdriver.Chrome(options=options, executable_path=r'C:\WebDrivers\chromedriver.exe')
  driver.get('https://duckduckgo.com/')
  WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.NAME, "q"))).send_keys("this is a search" + Keys.RETURN)
  while True:
        try:
            WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "a.result--more__btn"))).click()
            print("Clicked on More Results button")
        except TimeoutException:
            print("No more More Results button")
            break
  driver.quit()
  

• Console Output:

  Clicked on More Results button
  Clicked on More Results button
  Clicked on More Results button
  Clicked on More Results button
  Clicked on More Results button
  No more More Results button
  

You can find a relevant discussion in How to extract the text from the search results of duckduckgo using Selenium Python