Search code examples
pythonhtmlselenium-webdriverweb-scrapingdynamic

Not getting the data from web page with Selenium (Python)


I'm trying to get a table from a dynamic webpage using selenium but it's giving me issues. This is the Python code:

from selenium import webdriver

url = 'https://draft.shgn.com/nfc/public/dp/788/grid' 
driver = webdriver.Chrome('C:\chromedriver_win32\chromedriver.exe')

global_dynamicUrl = "https://draft.shgn.com/nfc/public/dp/788/grid"
driver.get(global_dynamicUrl)
table_area = driver.find_element("xpath", '//*[@id="result"]/table')
table_entries = table_area.find_elements_by_tag_name("tr")
print(len(table_entries))
driver.close()

But this produces a "NoSuchElementException" Error.

What am I doing wrong?

Thanks in advance.


Solution

  • There is no element with an ID of "result" on the page so that's why your locator isn't working. There is only one TABLE tag on the page so you can simplify your code down to

    from selenium import webdriver
    from selenium.webdriver.common.by import By
    from selenium.webdriver.support.wait import WebDriverWait
    from selenium.webdriver.support import expected_conditions as EC
    
    global_dynamicUrl = "https://draft.shgn.com/nfc/public/dp/788/grid"
    driver.get(global_dynamicUrl)
    wait = new WebDriverWait(driver, 10)
    table_entries = wait.until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "table tr"))
    print(len(table_entries))
    driver.close()