Infinite scrolling of webpage using Selenium Python

Question

I want to scroll till all the element is loaded. My code works well for starting till 7 8 pages using this code.

driver.get('https://www.bigbasket.com/ps/?q=rice')
last_height = driver.execute_script("return document.body.scrollHeight")

while True:
    # Scroll down to bottom
    driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
    # Wait to load page
    WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, '//div[@class="show-more"]/button')))
    driver.find_element_by_xpath('//div[@class="show-more"]/button').click()
    # Calculate new scroll height and compare with last scroll height
    new_height = driver.execute_script("return document.body.scrollHeight")
    if new_height == last_height:
        break
    last_height = new_height

The last height became equal to new height even though not all the element is loaded. I need to scroll till all element is loaded.

Answer 1

To scroll down till the bottom of the page and keep on clicking the Show More button you can use the following locator strategy:

• Code block:

  driver.get("https://www.bigbasket.com/ps/?q=rice")
  while True:
    try:
        driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
        print("Browser scrolled")
        driver.execute_script("arguments[0].click();", WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, "//button[text()='Show More']"))))
        print("Show More button clicked")
        time.sleep(3)
    except WebDriverException:
        print("No more scrolling")
        break
  

• Console Output:

  Browser scrolled
  Show More button clicked
  Browser scrolled
  Show More button clicked
  Browser scrolled
  Show More button clicked
  ...
  ...