Search code examples
pythonseleniumweb-scrapingdropdownwebdriverwait

UnexpectedTagNameException: Message: Select only works on <select> elements, not on <ul> error selecting an Dropdown option using Selenium and Python

The objective is to select either one of the four sub menus (i.e., Subject area, Title, Publisher, ISSN) as depicted in the picture below from the Scopus website that accessible via the link: https://www.scopus.com/sources?zone=TopNavBar&origin=NO%20ORIGIN%20DEFINED

enter image description here

The html snippet for the search result comb drop menu by the class name ui-menu ui-corner-bottom ui-widget ui-widget-content is

<ul aria-hidden="false" aria-labelledby="srcResultComboDrp-button" id="srcResultComboDrp-menu" role="listbox" tabindex="0" class="ui-menu ui-corner-bottom ui-widget ui-widget-content" aria-activedescendant="ui-id-1" aria-disabled="false" style="width: 251px;">
  <li class="ui-menu-item">
    <div id="ui-id-1" tabindex="-1" role="option" class="ui-menu-item-wrapper ui-state-active">Subject area</div>
  </li>
  <li class="ui-menu-item">
    <div id="ui-id-2" tabindex="-1" role="option" class="ui-menu-item-wrapper">Title</div>
  </li>
  <li class="ui-menu-item">
    <div id="ui-id-3" tabindex="-1" role="option" class="ui-menu-item-wrapper">Publisher</div>
  </li>
  <li class="ui-menu-item">
    <div id="ui-id-4" tabindex="-1" role="option" class="ui-menu-item-wrapper">ISSN</div>
  </li>
</ul>

Say we are interested to select the sub-menu Title, then the objective can be achieved as suggested by OP1, by the following lines;

from selenium import webdriver

driver = webdriver.Chrome(r"C:Browsers\chromedriver.exe")
url = 'https://www.scopus.com/sources?zone=TopNavBar&origin=NO%20ORIGIN%20DEFINED'
driver.get(url)
driver.find_element_by_xpath('//*[@id="ui-id-2"]').click()

However, the compiler return the following error;

Unable to locate element: {"method":"xpath","selector":"//*[@id="ui-id-2"]"}

Similarly, using the following line as suggested by OP2

from selenium import webdriver
from selenium.webdriver.support.ui import Select

driver = webdriver.Chrome(r"C:Browsers\chromedriver.exe")
url = 'https://www.scopus.com/sources?zone=TopNavBar&origin=NO%20ORIGIN%20DEFINED'
driver.get(url)
my_select = Select(driver.find_element_by_id('srcResultComboDrp-menu'))
my_select.select_by_visible_text('Title')

Return the following error:

selenium.common.exceptions.UnexpectedTagNameException: Message: Select only works on <select> elements, not on <ul>

May I know where did I do wrong? Appreciate for any help

Solution

  • To select either of the four sub menus among Subject area, Title, Publisher and ISSN as the items are within child <div> tags of their parent <li> tag, you have to induce WebDriverWait for the element_to_be_clickable() and you can use the following Locator Strategies:

    • Selecting Title using XPATH:

      driver.get("https://www.scopus.com/sources?zone=TopNavBar&origin=NO%20ORIGIN%20DEFINED")
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//span[@class='ui-selectmenu-text' and text()='Subject area']"))).click()
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//ul[@id='srcResultComboDrp-menu']//li[@class='ui-menu-item']/div[text()='Title']"))).click()

    • Note : You have to add the following imports :

      from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

    Reference

    You can find a couple of relevant detailed discussions in: