Search code examples
pythonseleniumxpathcss-selectorswebdriverwait

How to search in a webpage using selenium despite using correct search id

I am trying to search a web page using selenium. Below is the HTML code snippet of search page.

<div class="cmp-globalsite-search ac_box">
     <input id="searchString" type="text" class="form-control searchString 
          ac_input" placeholder="Search" aria-label="Search" required="" autocomplete="off">
     <input type="hidden" id="searchTargetUrl" value="/en/search">
     <a class="searchHead btn" href="#">
     <span class="nav-icon gcom-icon-search"></span>
     </a>
</div>

The above one is taken from https://www.gartner.com.

I am using below python code to use the search page.

url = 'https://www.gartner.com/'
driver = webdriver.Chrome(executable_path='path to chrome web driver')
driver.get(url)
elem = driver.find_element_by_id("SearchString")
elem.send_keys("Risk Management Solution")
elem.send_keys(keys.RETURN)

Strange thing is I am getting the below error:

NoSuchElementException: no such element: Unable to locate element: {"method":"css 
selector","selector":"[id="SearchString"]"}
(Session info: chrome=97.0.4692.71)

I have tried below as well:

elem = driver.find_element_by_xpath("""//input[@id="searchTargetUrl"]/input[@value = "Risk Management Solution""")

Getting similar error:

InvalidSelectorException: invalid selector: Unable to locate an element with the xpath 
expression //input[@id="searchTargetUrl"]/input[@value = "Risk Management Solution because 
of the following error:
SyntaxError: Failed to execute 'evaluate' on 'Document': The string 
'//input[@id="searchTargetUrl"]/input[@value = "Risk Management Solution' is not a valid 
XPath expression.
(Session info: chrome=97.0.4692.71)

I am novice to selenium and therefore asking for any clue.

Update

I have copied the full xpath of the search page from the above site.

//*[@id="gartnerinternalpage-9ac4556e05"]/div[2]/div/div/div/div/div/div/section/div/div/div[1]/div[2]/div/div[1]/input

So I had replaced the above xpath string in the python code and added //input[@value={0}]

Still no luck!!

Solution

  • The Search field is not visible by default, to send a character sequence to the search field you need to click() on the search icon first inducing WebDriverWait for the element_to_be_clickable() and you can use either of the following Locator Strategies:

    • Using CSS_SELECTOR:

      driver.get("https://www.gartner.com/en")
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button#onetrust-accept-btn-handler"))).click()
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "span.nav-icon.gcom-icon-search"))).click()
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "input#searchString"))).send_keys("Risk Management Solution" + Keys.RETURN)

    • Using XPATH:

      driver.get("https://www.gartner.com/en")
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//button[@id='onetrust-accept-btn-handler']"))).click()
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//span[@class='nav-icon gcom-icon-search']"))).click()
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//input[@id='searchString']"))).send_keys("Risk Management Solution" + Keys.RETURN)

    • Note: You have to add the following imports :

      from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

    • Browser Snapshot:

    gartner_com