I am trying to scrape the list of companies from a dropdown in the website https://www.goodjobsfirst.org/violation-tracker , the drop-down is the parent company one.
I am running the following code
from selenium.webdriver.support.ui import Select
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
browser = webdriver.Chrome(executable_path=r"C:\webdrivers\chromedriver.exe")
url = ('https://www.goodjobsfirst.org/violation-tracker')
browser.get(url)
browser.maximize_window()
element = WebDriverWait(browser, 20).until(Select(browser.find_element_by_id("edit-field-violation-parent-value")))
browser.quit()
but for I get the error
NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":"[id="edit-field-violation-parent-value"]"}
(Session info: chrome=83.0.4103.61)
I checked different times and edit-field-violation.. is indeed the ID. I also tried to get it using the Xpath and other selectors but without luck. What am I getting wrong? How can I get the list of all the companies in the drop-down?
Thanks!
This element is inside <iframe> and Selenium treads frame like separated content/page and you have to use browser.switch_to.frame(...) to access it.
from selenium import webdriver
browser = webdriver.Chrome(executable_path=r"C:\webdrivers\chromedriver.exe")
#browser = webdriver.Firefox()
url = ('https://www.goodjobsfirst.org/violation-tracker')
browser.get(url)
browser.maximize_window()
frame = browser.find_element_by_tag_name("iframe")
print('frame:', frame)
browser.switch_to.frame(frame)
element = browser.find_element_by_id("edit-field-violation-parent-value")
print('element:', element)
select = Select(select)
print('options number:', len(select.options))
for number, item in enumerate(select.options, 1):
print(number, item.text)
# go back to main content
#browser.switch_to.default_content()
browser.quit()