Scraping with Selenium returning empty values after first elements

Question

I'm scraping a webpage and for some reason it returns correctly the first 12 elements and not the remaining 24, for a total of 36 shown in the page.

search_names = driver.find_elements_by_class_name('offerList-item-description-title')
names = []
for name in search_names:
    names.append(name.text)

search_names has a length of 36, but it returns the following (Sample):

[1 , 2, 3 , 4 , 5 , 6 , 7 , 8 , 9 , 10 , 11 , 12 , '', '', ... , '']

Any idea on why this might be happening?

Here's a snippet of the source code:

Answer 1

To extract the texts from all of the elements with class as offerList-item-description-title using Selenium and python you have to induce WebDriverWait for visibility_of_all_elements_located() and you can use either of the following Locator Strategies:

• Using CLASS_NAME:

  print([my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CLASS_NAME, "offerList-item-description-title")))])
  

• Using CSS_SELECTOR:

  print([my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "div.offerList-item-description-title")))])
  

• Using XPATH:

  print([my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//div[@class='offerList-item-description-title']")))])
  

• Note : You have to add the following imports :

  from selenium.webdriver.support.ui import WebDriverWait
  from selenium.webdriver.common.by import By
  from selenium.webdriver.support import expected_conditions as EC