Search code examples
pythonselenium-webdriverxpathselenium-chromedriver

Using xpath and selenium to select a HTML element in Python

Thanks for letting me ask my questions here. I am new to Selenium and XPath, just trying to scrape a not-so-simple website using Python.

My questions are:

  1. Do you have an answer to my specific question on how to select the HTML element in question?
  2. Do you have any addition to the learning ressources I listed below (which seem very helpful, but I was seemingly not yet advanced enough to apply them to my situation)?

The specific question: I have a HTML file that looks as follows and want to extract the 'data-testid="qs-select-make"' element (in the end, I want to use selenium to update the drop-down menu)

For the life of me, I do not get it to work....

<div class = "a">
<div class = "ab">
<div class = "abc">
<div class = "abcd">
<select class="tya6p HaBLt A4yQa q0MnL" placeholder="Any" data-testid="qs-select- 
make"><option selected="" value="">Any</option>

Using the google chrome web developer I already found the "correct" (albeit not very nice path) seems to be [@id="root"]/div/div/article[1]/section/div/div[2]/div[1]/div[1]/div/select

Still, the following code, attempting to insert the make "Audi" into the drop-down menu, fails with an Invalid Selector Exception:

make_string = "//select[//* 
[@id='root"]/div/div/article[1]/section/div/div[2]/div[1]/div[1]/div/select]option 
selected[text()='{}']".format("Audi") 

driver.find_element("xpath", make_string).click() #use selenium to click the button

Does anyone know what I am doing wrong, and of a nicer way to do it?

Regarding question 2, the resource. So far I have used:

  • Stackoverflow, helpful as always, in particular this question
  • A very useful testsigma blogpost, which pointed me towards the chrome browser web developer to at least the the path:here
  • The Selenium documentation - very nicely written, but since I am beginner I am not yet able to apply the general concepts to my specific issue... apologies: Here
Solution

  • Use the below xpath to identify the element. Please add syncronisation time as well.

    driver.find_element("xpath", "//select[@data-testid='qs-select-make']//option[text()='{}']".format("Audi")).click()