I am trying to extract a newspaper data by clicking at next button to get more links

Question

I am trying to extract business standard newspaper economy section data by clicking on the links but I am failing to do it.

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
# Here Chrome is used

# URL of website
url = "https://www.business-standard.com/search?q=economy"

# Opening the website
driver.get(url)

# #button = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//a[@href='#']"))).click()
button = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//div[contains(., 'Next')]]"))).click()

Answer 1

There are several issues here:

1. You need to close the floating banner
2. You are using a wrong locator.
3. there is no need to define button element on the left side when you click the element returned instantly.

This should work better:

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
# Here Chrome is used

# URL of website
url = "https://www.business-standard.com/search?q=economy"

# Opening the website
driver.get(url)

wait = WebDriverWait(driver, 20)
wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, "div.btnno"))).click()
wait.until(EC.element_to_be_clickable((By.XPATH, "//div[@class='next-colum']"))).click()