I am scraping true car using selenium and python. I currently have the code set up to find the unordered list containing all listings and then iterate over them to print the price. Using the local xpath will only ever print the price value of the first element in the unordered list. I tried dynamically updating the xpath in the for loop such that the index in the xpath will change but then I get an error that it cannot find the element. Here is the code I have to better demonstrate this.
driver = webdriver.Chrome(ChromeDriverManager().install())
driver.get("https://www.truecar.com/used-cars-for-sale/listings/bmw/m4/location-palm-desert-ca/")
listingSection = driver.find_element(By.XPATH, '/html/body/div[3]/div/div[2]/main/div/div[2]/div[1]/div[2]/div[2]/div[2]/ul')
listings = listingSection.find_elements(By.TAG_NAME, "li")
for i, listing in enumerate(listings):
// wrong element
price = listing.find_element(By.XPATH, "//*[@id="mainContent"]/div/div[2]/div[1]/div[2]/div[2]/div[2]/ul/li[1]/div/div/div[3]/div[2]/div/div[2]/div/div").text //this will only return the first item in the list and sadly the list elements do not have ids
// error
price = listing.find_element(By.XPATH, f"//*[@id="mainContent"]/div/div[2]/div[1]/div[2]/div[2]/div[2]/ul/li[{i + 1}]/div/div/div[3]/div[2]/div/div[2]/div/div").text // this returns an error
print(price)
I know that the reason for this is likely that the id field for the xpath is not relative to the list elements themselves but since they have no ids I am not sure how to go about this as I am new to webscraping.
They don't have ids but they do have data-test attribute, which I suspect was added for the purpose of Selenium testing.
listings = driver.find_elements(By.CSS_SELECTOR, '[data-test="vehicleCardPricingBlockPrice"]')
for listing in listings:
print(listing.text)
As a side not, when using WebElement to locate another WebElement with xpath you need to tell it to use current context with .
listing.find_element(By.XPATH, './/*[@id="mainContent"]')