Search code examples
pythonseleniumwebdriverwait

How do I loop 'WebDriverWait' function when it cannot find an information?


How do I repeat(loop) the below code even not available 'URL' case?

For example, I do have excel with many URLs of music on Discogs.com.

https://www.discogs.com/master/561393-The-High-School-Musical-Cast-High-School-Musical-2-Soundtrack

https://www.discogs.com/master/1338438-Zedd-Maren-Morris-Grey-The-Middle

https://www.discogs.com/master/606957-Zendaya-Zendaya

...

However, some row does not have the exact information that I wished for. (genres..) Currently, the code shows "NoSuchWindowException... errors..' How do I just ignore the not-available URL and repeat the next loop?

driver.get(url)
  genre = None
  try:
    WebDriverWait(driver, 5).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button#onetrust-accept-btn-handler"))).click()
    genre = [genre.text for genre in WebDriverWait(driver, 3).until(EC.visibility_of_all_elements_located((By.XPATH, "//tr/th[@scope='row' and contains(., 'Genre')]//following::td[1]//a")))]
  except selenium.common.exceptions.NoSuchElementException:

 if genre != None:
    song_url_dict_excel.iloc[i,2] = genre


Solution

  • To get rid of None type error, you can apply if else None statement. Try this:

    from selenium.webdriver.common.by import By
    from selenium.webdriver.support import expected_conditions as EC
    from selenium.webdriver.support.ui import WebDriverWait
    from selenium import webdriver
    from webdriver_manager.chrome import ChromeDriverManager
    
    driver = webdriver.Chrome(ChromeDriverManager().install())
    
    url = 'https://www.discogs.com/master/561393-The-High-School-Musical-Cast-High-School-Musical-2-Soundtrack'
    
    
    
    driver.get(url)
    driver.maximize_window()
    wait =  WebDriverWait(driver, 10)
    
    button = wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, 'button#onetrust-accept-btn-handler'))).click()
    genre = wait.until(EC.visibility_of_all_elements_located((By.XPATH, '(//*[@class="table_1fWaB"]/tbody/tr)[1]/td//a')))
    for gen in genre:
        gen = gen.text if gen else None
        print(gen)
    

    Output:

    Electronic
    Rock
    Pop
    Stage & Screen