Search code examples
pythonselenium-webdriverweb-scraping

Python Selenium - 'Unable to locate element' after made visible

I need your help. I'm trying to scrape some data from tripadvisor using Selenium in Python 2.7. However, I'm getting stuck at one point.

After browsing to the correct page, I'm trying to filter the hotels on certain prices. To do this, you do a mouse over or click on 'price' and then select the appropiate value like (€3 - € 13).

After clicking on price and then the value. I'm getting the error that the element is not visible or unable to locate, while it is clearly visible.

code

from urllib import urlopen
import time
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

city = 'nha thrang'


url = 'http://www.tripadvisor.nl/Hotels'

driver = webdriver.Firefox()

# open browser
driver.get(url)
time.sleep(5)

# insert city & dates
driver.find_element_by_id('searchbox').send_keys(city)
driver.find_element_by_id('date_picker_in_188616').click()
driver.find_elements_by_class_name('day')[15].click()
driver.find_element_by_id('date_picker_out_188616').click()
driver.find_elements_by_class_name('day')[16].click()

time.sleep(5)

# click search
driver.find_element_by_id('SUBMIT_HOTELS').click()

# close popup
time.sleep(5)
try:
    driver.switch_to.window(driver.window_handles[1])
    driver.close()
    driver.switch_to.window(driver.window_handles[0])
except:
    ''

# click on 'price'. Works!
driver.find_element_by_xpath('//div[starts-with(@class, "JFY_hotel_filter_icon enabled price sprite-price")]').click()

# click on particular price. doesn't work.
driver.find_element_by_xpath('//div[starts-with(@class, "jfy_tag_style jfy_filter_p_4 jfy_cloud")]').click()

Error

Traceback (most recent call last):
File "<pyshell#30>", line 1, in <module>
driver.find_element_by_xpath('//div[starts-with(@class, "jfy_tag_style jfy_filter_p_4 jfy_cloud")]').click()
File "C:\Python27\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 230, in find_element_by_xpath
return self.find_element(by=By.XPATH, value=xpath)
File "C:\Python27\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 662, in find_element
{'using': by, 'value': value})['value']
File "C:\Python27\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 173, in execute
self.error_handler.check_response(response)
File "C:\Python27\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 166, in check_response
raise exception_class(message, screen, stacktrace)
NoSuchElementException: Message: Unable to locate element: {"method":"xpath","selector":"//div[starts-with(@class, \"jfy_tag_style jfy_filter_p_4 jfy_cloud\")]"}
Stacktrace:
at FirefoxDriver.prototype.findElementInternal_ (file:///c:/users/j6057~1.kro/appdata/local/temp/tmpdgovsc/extensions/[email protected]/components/driver-component.js:9641:26)
at FirefoxDriver.prototype.findElement (file:///c:/users/j6057~1.kro/appdata/local/temp/tmpdgovsc/extensions/[email protected]/components/driver-component.js:9650:3)
at DelayedCommand.prototype.executeInternal_/h (file:///c:/users/j6057~1.kro/appdata/local/temp/tmpdgovsc/extensions/[email protected]/components/command-processor.js:11635:16)
at DelayedCommand.prototype.executeInternal_ (file:///c:/users/j6057~1.kro/appdata/local/temp/tmpdgovsc/extensions/[email protected]/components/command-processor.js:11640:7)
at DelayedCommand.prototype.execute/< (file:///c:/users/j6057~1.kro/appdata/local/temp/tmpdgovsc/extensions/[email protected]/components/command-processor.js:11582:5)
Solution

  • You need to apply multiple changes to make it work:

    Working code (selecting "USD 25 - 50" range):

    from selenium import webdriver
from selenium.webdriver import ActionChains
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC


city = 'nha thrang'

url = 'http://www.tripadvisor.nl/Hotels'
driver = webdriver.Chrome()
driver.get(url)

# insert city & dates
searchbox = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.ID, 'searchbox')))
searchbox.send_keys(city)

driver.find_element_by_xpath('//span[starts-with(@id, "date_picker_in_")]').click()
driver.find_elements_by_class_name('day')[15].click()

driver.find_element_by_xpath('//span[starts-with(@id, "date_picker_out_")]').click()
driver.find_elements_by_class_name('day')[16].click()

# click search
driver.find_element_by_id('SUBMIT_HOTELS').click()

# select price range
price = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.XPATH, '//div[starts-with(@class, "JFY_hotel_filter_icon enabled price sprite-price")]')))

ActionChains(driver).move_to_element(price).perform()

price_range = WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.XPATH, '(//div[contains(@class, "jfy_filter_bar_price")]//div[@value="p 8"])[last()]')))
price_range.click()

    Results into:

    enter image description here