Search code examples
pythonselenium-webdriverweb-scraping

ElementNotVisibleException: Message: element not visible - Python3 Selenium

I have been tasked with writing a parser to click a href link, that looks like a button, on a website and I am having some issues.

Here's the html: https://pastebin.com/HDKLXpdJ

Here's the source html: https://pastebin.com/PgT91kJs

Python code:

browser = webdriver.Chrome()
...
try:

    element = WebDriverWait(browser, 20).until(
        EC.presence_of_element_located((By.ID, "reply-panel-reveal-btn")))

finally:
      elem = browser.find_element_by_xpath("//A[@id='reply-panel-reveal-btn']").click()

I am getting this error.

selenium.common.exceptions.ElementNotVisibleException: Message: element not visible

I have tried switching between ChromeDriver and GeckoDriver(FF), but I'm getting the same error, over and over again. I even tried waiting for 10 secs to load, same results.

Full error text:

File "C:/Users/DEM/PycharmProjects/Test/Scrape.py", line 46, in <module> elem = browser.find_element_by_xpath("//A[@id='reply-panel-reveal-btn']").click()
File "C:\Users\DEM\AppData\Local\Programs\Python\Python36-32\lib\site-packages\selenium\webdriver\remote\webelement.py", line 77, in click self._execute(Command.CLICK_ELEMENT)
File "C:\Users\DEM\AppData\Local\Programs\Python\Python36-32\lib\site-packages\selenium\webdriver\remote\webelement.py", line 493, in _execute return self._parent.execute(command, params)
File "C:\Users\DEM\AppData\Local\Programs\Python\Python36-32\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 256, in execute self.error_handler.check_response(response)
File "C:\Users\DEM\AppData\Local\Programs\Python\Python36-32\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 194, in check_response raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.ElementNotVisibleException: Message: element not visible
(Session info: chrome=61.0.3163.100)
(Driver info: chromedriver=2.33.506120 (e3e53437346286c0bc2d2dc9aa4915ba81d9023f),platform=Windows NT 6.1.7601 SP1 x86_64)

Video link on how it should work :

https://streamable.com/e1uvm

Edit:

Problem solved, check @JeffC answer.

The correct code :

browser = webdriver.Chrome()
...
element = WebDriverWait(driver, 20).until(
EC.element_to_be_clickable((By.XPATH, "(//a[@id='reply-panel-reveal-btn'])[2]")));
element.click()

The problem :

I was waiting for the presence of the element. Presence doesn't mean that the element is visible or clickable, it just means the element is in DOM. Also, I was waiting for the first element, which happens to be invisible. I needed to locate the second element and just wait for it to be clicked.

Solution

  • There are a couple issues.

    1. You are waiting for presence of the element. Presence just means that the element is in the DOM, not that it's visible or clickable. If you are going to wait and click an element, wait for it to be clickable. If you are going to wait to send_keys() or get the text from an element, wait for it to be visible. The are some uses for presence but I don't use it often. Having said that...

    2. There are two elements that match your locator, id=reply-panel-reveal-btn. The first one that match just happens to be invisible. With XPath we can create a locator that finds the second element, wait for it to be clickable, and then click it.

      element = WebDriverWait(driver, 20).until(
    EC.element_to_be_clickable((By.XPATH, "(//a[@id='reply-panel-reveal-btn'])[2]")));
element.click()