Search code examples
pythonseleniumweb-scrapingtry-catchwebdriverwait

How to scroll the element until a certain word appears?


I'm scraping Google Maps and I need to know how to scroll the query column until the word appears

"You've reached the end of the list".
I am using selenium for scraping.
Code I currently use:

for a in range(100):
    barraRolagem = wait.until(EC.presence_of_element_located((By.XPATH, 
    "//div[@role='main']//div[contains(@aria-label,'" + procurar + "')]")))
     driver.execute_script("arguments[0].scroll(0, arguments[0].scrollHeight);", barraRolagem)

this code works, however the range is variable and is not certain, that is, it can scroll a lot and it can also scroll a little, for this reason I need to stop scrolling when the code finds the phrase

"You've reached the end of the list"

enter image description here

Link

https://www.google.com.br/maps/search/contabilidade+balneario+camboriu/@-26.9905418,-48.6289914,15z


Solution

  • You can scroll in a loop until "You've reached the end of the list" text is visible.
    When text is found visible - break the loop. Otherwise do a scroll.
    Since in case element not visible exception is thrown try-except block is needed here.
    Additional scroll is added after the element is found visible since Selenium detects that element visible while it is still not actually visible, so one more scroll should be done finally.
    The following code works correctly:

    import time
    
    from selenium import webdriver
    from selenium.webdriver.chrome.service import Service
    from selenium.webdriver.chrome.options import Options
    from selenium.webdriver.support.ui import WebDriverWait
    from selenium.webdriver.common.by import By
    from selenium.webdriver.support import expected_conditions as EC
    
    options = Options()
    options.add_argument("start-maximized")
    options.add_argument('--disable-notifications')
    
    webdriver_service = Service('C:\webdrivers\chromedriver.exe')
    driver = webdriver.Chrome(options=options, service=webdriver_service)
    wait = WebDriverWait(driver, 1)
    
    url = "https://www.google.com.br/maps/search/contabilidade+balneario+camboriu/@-26.9905418,-48.6289914,15z"
    driver.get(url)
    while True:
        try:
            wait.until(EC.visibility_of_element_located((By.XPATH, "//span[contains(text(),'reached the end')]")))
            barraRolagem = wait.until(EC.presence_of_element_located((By.XPATH, "//div[@role='main']//div[@aria-label]")))
            driver.execute_script("arguments[0].scroll(0, arguments[0].scrollHeight);", barraRolagem)
            break
        except:
            barraRolagem = wait.until(EC.presence_of_element_located((By.XPATH, "//div[@role='main']//div[@aria-label]")))
            driver.execute_script("arguments[0].scroll(0, arguments[0].scrollHeight);", barraRolagem)
            time.sleep(0.5)