Search code examples
pythonseleniumgoogle-chromeselenium-webdriverlistitem

How to select a g-menu-item in selenium (Python)

I am trying to find news articles related to different web searches within a custom time frame. If someone knows a better way to do this than with selenium then please do tell. I tried to use API's but they are all expensive. My Code:

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
driver = webdriver.Chrome(executable_path=r".\chromedriver.exe")
driver.get("https://www.google.com")
cookie_button = driver.find_element_by_id('L2AGLb')
cookie_button.click()
search_bar = driver.find_element_by_name("q")
search_bar.clear()
search_bar.send_keys("Taylor Swift")
search_bar.send_keys(Keys.RETURN)
news_button = driver.find_element_by_css_selector('[data-hveid="CAEQAw"]')
news_button.click()
tools_button = driver.find_element_by_id('hdtb-tls')
tools_button.click()
recent_button = driver.find_element_by_class_name('KTBKoe')
recent_button.click()
custom_range_button = dashboards_button = driver.find_elements_by_tag_name('g-menu-item')[6]
custom_range_button.click()
driver.close()

I am trying to click this button: Image of google chrome

I have tried to select, the span, all parent divs and now the g-menu-item itself, in every case I have received the error:

selenium.common.exceptions.ElementNotInteractableException: Message: element not interactable

I understand this can happen when the element is not visible or can not be clicked but I also tried adding in sleep(2) between each operation, and I can see in the browser that selenium opens that it gets to the point in the image I have sent and then can't find the custom range button ??

In short: Do you know how I can click this custom range button, Or do you know how to scrape a lot of news articles from specific dates (without expensive API's)

Solution

  • Those g-menu are special type of tag,

    you can locate them using :

    //*[name()='g-menu-item']

    this xpath will point to all the nodes.

    further, I see that you are interested in 6th item.

    (//*[name()='g-menu-item'])[6]

    in code :

    custom_range_button = driver.find_element_by_xpath("(//*[name()='g-menu-item'])[6]")
custom_range_button.click()