Selenium Python: Unable to scroll and click on Load more stories button

Question

I am new to Selenium and would appreciate advice. I am trying to click on the load more stories button on NPR's website (https://www.npr.org/sections/news/). My code seems good, but when I run it more stories are never loaded. I don't get an error at the same time the code just doesn't behave as expected.

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

driver = webdriver.Firefox(executable_path='/home/myname/projects/myproject/geckodriver')
# implicit wait for 5 seconds
driver.implicitly_wait(5)
#driver.maximize_window()
driver.get('https://www.npr.org/sections/news/')

element = driver.find_element(By.XPATH, '/html/body/main/div[2]/section/div[7]/button')
driver.execute_script("arguments[0].scrollIntoView();", element)
driver.execute_script("arguments[0].click();", element)

Answer 1

The webpage npr have a infinitescrollwrap. As a demonstration to click() thrice on Load more stories you can use the following Locator Strategies:

• Code Block:

  driver.get("https://www.npr.org/sections/news/")
  WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button.global-modal__dismiss"))).click()
  for i in range(3):
    driver.execute_script("arguments[0].scrollIntoView();", WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "button.options__load-more"))))
    WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button.options__load-more"))).click()
    print("Load more stories clicked")
  

• Console Output:

  Load more stories clicked
  Load more stories clicked
  Load more stories clicked
  

• Note : You have to add the following imports :

  from selenium.webdriver.support.ui import WebDriverWait
  from selenium.webdriver.common.by import By
  from selenium.webdriver.support import expected_conditions as EC