Selenium XPath: How to click on the Load More button

Question

I am trying to scrape off-campus apartment names and addresses for an apartment listing website for universities. My current obstacle is clicking a "Load More" Button in selenium that can appear x number of times per search result. I am new to webscraping, and I was wondering if someone could help me click this button until it is gone, so I then can gather all of the other information using beautifulsoup.

Website: https://www.rentcollegepads.com/off-campus-housing/mississippi-state/search

Answer 1

To click on the element Load More you need to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following locator strategies:

• Using CSS_SELECTOR:

  WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button.btn.btn-success.btn-block > i.fa.fa-refresh"))).click()
  

• Using XPATH:

  WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//button[@class='btn btn-success btn-block' and contains(., 'Load More')]"))).click()
  

• Note : You have to add the following imports :

  from selenium.webdriver.support.ui import WebDriverWait
  from selenium.webdriver.common.by import By
  from selenium.webdriver.support import expected_conditions as EC