Search code examples
selenium-webdriverweb-scrapingxpathcss-selectorswebdriverwait

Unable to get an element using Selenium

I have been unable to get the rating number from the website https://www.hoka.com/en/us/mens-everyday-running-shoes/rincon-3/1119395.html using selenium.

This is what I have as Python code:

    from selenium import webdriver
    from selenium.webdriver.common.by import By

    url = "https://www.hoka.com/en/us/mens-everyday-running-shoes/rincon-3/1119395.html"
    driver = webdriver.Chrome()
    driver.get(url)
    res = driver.find_elements(By.TAG_NAME, 'h3')
    for i in res:
        print(i.text)

I am getting empty result.

Here's the line with the info I want to extract:

Here's the line with the info I want to extract

Solution

  • To print the text 4.4 ideally you need to induce WebDriverWait for the presence_of_element_located() and you can use either of the following locator strategies:

    • Using CSS_SELECTOR and text attribute:

      driver.get(url='https://www.hoka.com/en/us/mens-everyday-running-shoes/rincon-3/1119395.html')
print(WebDriverWait(driver, 20).until(EC.presence_of_element_located((By.CSS_SELECTOR, "h3.pr-review-snapshot-snippets-headline"))).text)

    • Using XPATH and get_attribute():

      driver.get(url='https://www.hoka.com/en/us/mens-everyday-running-shoes/rincon-3/1119395.html')
print(WebDriverWait(driver, 20).until(EC.presence_of_element_located((By.XPATH, "//h3[starts-with(@class, 'pr-review-snapshot-snippets-headline')]"))).get_attribute("innerHTML"))

    • Console Output:

      4.4

    • Note : You have to add the following imports :

      from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

    You can find a relevant discussion in How to retrieve the text of a WebElement using Selenium - Python