Search code examples
pythonseleniumselenium-webdriverweb-scrapingwebdriver

Cannot find xpath element to click with webdriver


I'm using the https://www.perlego.com/ website to do some scraping for books, my current problem is that i'm trying to search using the isbn Number of the book so the webdriver gets this following page for example https://www.perlego.com/search?query=9780717183241

However, using page inspection on the website, i cannot find the element that i can click on using webdriver

image of page inspection

the idea is to click on the book and load the page that comes with it


Solution

  • The following code closes the cookies banner and clicks on the book link and opens it

    from selenium import webdriver
    from selenium.webdriver.chrome.service import Service
    from selenium.webdriver.chrome.options import Options
    from selenium.webdriver.support.ui import WebDriverWait
    from selenium.webdriver.common.by import By
    from selenium.webdriver.support import expected_conditions as EC
    
    options = Options()
    options.add_argument("start-maximized")
    
    webdriver_service = Service('C:\webdrivers\chromedriver.exe')
    driver = webdriver.Chrome(options=options, service=webdriver_service)
    wait = WebDriverWait(driver, 10)
    
    url = "https://www.perlego.com/search?query=9780717183241&tab=book&page=1&language=All%20languages&publicationDate=&topic=&publisher=&author=&format="
    
    driver.get(url)
    wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button[data-testid*='Cookies']"))).click()
    wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "a[data-test-locator*='new-book']"))).click()