Search code examples
pythongoogle-chromeseleniumheadless

Headless chrome with selenium, can only find ways to scroll non-headless


There is a lot to find about this subject, but can't figure this out. I need to scroll to the end of the page of a (not so long) infinity scroll. I have 2 options that work with chrome non-headless but doesn't seem to work headless.

The first one which i liked the most, works beautiful and found here on SA:

driver = webdriver.Chrome('c:/cd.exe', chrome_options=chrome_options)
driver.get('http://www.website.com')

while True:
    count = len(driver.find_elements_by_xpath('//div[@itemprop="itemListElement"]'))
    print(count)
    driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
    try:
        WebDriverWait(driver, 50).until(EC.visibility_of_element_located((By.XPATH,
                                                                          "//div[@itemprop='itemListElement'][%s]" % str(count + 1))))
    except TimeoutException:
        break

Second more hack job after realizing i can't get away with above in headless mode:

driver = webdriver.Chrome('c:/cd.exe', chrome_options=chrome_options)
driver.get('https://www.website.com')

while True:

    count = len(driver.find_elements_by_xpath('//div[@itemprop="itemListElement"]'))
    actions = ActionChains(driver)
    actions.send_keys(Keys.PAGE_DOWN)
    actions.perform()
    actions.send_keys(Keys.PAGE_DOWN)
    actions.perform()


    # focus_element_scroll = driver.find_elements_by_xpath('//section[@class="occasion-content"]')
    # driver.find_elements_by_xpath('//div[@itemprop="itemListElement"]')[-1].send_keys(Keys.PAGE_DOWN)
    # driver.find_elements_by_xpath('//div[@itemprop="itemListElement"]')[-1].send_keys(Keys.PAGE_DOWN)
    # self.driver.find_element_by_css_selector("ul.list-with-results").send_keys(Keys.ARROW_DOWN)
    print(count)
    # driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")

    try:  
        WebDriverWait(driver, 50).until(EC.visibility_of_element_located((By.XPATH,
                                                                          "//div[@itemprop='itemListElement'][%s]" % str(count + 1))))
    except TimeoutException:
        break

So both work in chrome but don't in headless mode, i need to push them to a ubuntu vps where they need to be headless, i know of the xvfb option but i am glad i could remove that and use native chrome since the droplets don't have much memmory.

Edit: Just tried this approach with focus on an element in the footer, also works in non-headless but doesn't in headless:

ActionChains(driver).move_to_element(focus[0]).perform()

Someone got a different approach?

Edit Just want to know if it is possible to scroll with chrome in headless mode!


Solution

  • Found the answer after 2 days trying different combinations of versions of selenium, chrome and chromedriver i alsmost gave up and wanted to go with the xvfb.

    Already tried to maximze the window in the chrome arguments, that didn't help. But this time i tried setting a manual window size. That helped.

        chrome_options.add_argument("window-size=1920,1080")
    

    Posting here so that the next one wont take as long as me.