Search code examples
pythonselenium-webdriverxpathcss-selectorsgetelementbyid

Webscraping an element with style="display: none;" using Python and Selenium Webdriver

I want to scrape web site for scheduling medical appointments in hospital in my country, for this purpose I use this code:

from selenium import webdriver

d = webdriver.Chrome()
d.get("http://www.mojtermin.mk/map/spetsijalist?specialty=meditsinska_genetika&location=grad_skopje&clinic=jzu_uk_za_ginekologija_i_akusherstvo&resource=873745910")
d.maximize_window()
d.implicitly_wait(20)
d.find_element_by_id('cdk-describedby-message-0').text
termins = d.find_element_by_id(id_='cdk-describedby-message-0').text

print(termins)

When I run code, Chrome is open successfully and show website, but find_element_by_id, didn't return anything only empty string.

when I inspect element in chrome for this element I find this :

<div id="cdk-describedby-message-container" aria-hidden="true" style="display: none;"><div id="cdk-describedby-message-0">Нема слободни термини во наредните 3 месеци.</div></div>

I expect find_element_by_id return value off the element, in this case :

"Нема слободни термини во наредните 3 месеци"

I tried many different ways but every time i get empty string or unable to find element.

Note: translate for value: "There are no free schedules in the next 3 months"

Solution

  • Use get_attribute("textContent") instead of text.

    Induce WebDriverWait() and wait for presence_of_element_located() and following css selector.

    from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

d=webdriver.Chrome()
d.get("http://www.mojtermin.mk/map/spetsijalist?specialty=meditsinska_genetika&location=grad_skopje&clinic=jzu_uk_za_ginekologija_i_akusherstvo&resource=873745910")
d.maximize_window()
print(WebDriverWait(d,20).until(EC.presence_of_element_located((By.CSS_SELECTOR,"#cdk-describedby-message-container>#cdk-describedby-message-0"))).get_attribute("textContent"))

    Output:

    Нема слободни термини во наредните 3 месеци.