Search code examples
pythonseleniumdojodijit.form

Selenium with Python : how to click dojo combobox values

I'm navigating a JS-heavy webpage with Selenium and I need to be able to interact with a dojo component on the page. The page I'm looking at has a dojo dijit form with a combobox that has subject names for my university. I want to expose and iteratively click on every item in the list in order to scrape the course names for that subject when it redirects. The list items are exposed when the dropdown arrow button is clicked.

the url I'm automating: http://sis.rutgers.edu/soc/#subjects?semester=12020&campus=NB,NK,CM&level=U,G

I'm inspecting the element for the dropdown button and copying the XPath. 

dropdownButton = driver.find_element_by_xpath('//*[@id="widget_dijit_form_FilteringSelect_0"]/div[1]/input')

Running this yields:

NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//*[@id="widget_dijit_form_FilteringSelect_0"]/div[1]/input"}

enter image description here

enter image description here

EDIT: I've made some progress, turn out the element wasn't rendered by the time find_by_xpath was called. I added a wait in my program, and now Selenium is able to locate and click the drowdown button.

Solution

  • Use WebDriverWait to wait require element conditions. Dropdown disappears on any action on the page, that's why to get option locator you can do one of the following:

    • all options loaded after first expand, that's why you can search option element by text in chrome dev tools and get locator
    • pause and inspect the element.

    You can google best practice for locators,here and here.

    from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

driver = webdriver.Chrome()
wait = WebDriverWait(driver, 10)

with driver:
    driver.get("http://sis.rutgers.edu/soc/#subjects?semester=12020&campus=NB,NK,CM&level=U,G")
    wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "#filteringSelectDiv .dijitArrowButtonInner"))).click()
    options = driver.execute_script('return [...arguments[0]].map(e=>e.textContent)',
                                    wait.until(EC.presence_of_all_elements_located(
                                        (By.CSS_SELECTOR, ".dijitComboBoxMenuPopup .dijitMenuItem[item]"))))

    for option in options:
        driver.find_element_by_css_selector(".dijitInputInner").clear()
        driver.find_element_by_css_selector(".dijitInputInner").send_keys(option, Keys.TAB)
        wait.until(lambda d: d.execute_script("return document.readyState === 'complete'"))
        # collect data