I am using the following code to collect all the reviews on a page, but it collects only the first one. However, the xpath used in the code highlights 10 elements when elements are inspected on the website.
import time
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
url = "https://www.tripadvisor.com/Attraction_Review-g262047-d568915-Reviews-Spiaggia_dei_Conigli-Lampedusa_Islands_of_Sicily_Sicily.html"
driver = webdriver.Firefox()
time.sleep(5)
driver.get(url)
time.sleep(5)
# accepts the cookie
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.ID, 'onetrust-accept-btn-handler'))).click()
# clicks to reach the review page
inputElement = driver.find_element(By.XPATH, '//a[@href="#REVIEWS"]/div/span').click()
time.sleep(5)
# attempt 1: does not catch all 10 reviews
#reivew_list = driver.find_element(By.XPATH, '//div[@style="line-break: normal; cursor: auto;"]/div/span')
# attempt 2: does not catch all 10 reviews
review_list = driver.find_element(By.XPATH, '//div[@data-automation="reviewCard"]//div[5]/div[1]/div/span')
for review in review_list:
rev = review.text
print(rev)
Please help me understand how to collect all the reviews on the webpage.
You are currently using driver.find_element() which returns exactly 1 matching element (the first found within DOM tree).
Just change it to driver.find_elements() which returns all matching elements.
An overview for the selenium python library: [1]: https://selenium-python.readthedocs.io/locating-elements.html