selenium.TimeoutException: Expected condition failed: waiting for element to be clickable with Selenium Webdriver

Question

I wanted to web scrape this website: mediamarkt computer website

Below is the code:

System.setProperty("webdriver.chrome.driver", "path/to/chromedriver/exe/worksFine");

ChromeDriverService service = new ChromeDriverService.Builder()
                .withLogOutput(System.out)
                .build();

WebDriver driver = new ChromeDriver(service);
driver.get("https://www.mediamarkt.at/de/category/alle-notebooks-884.html");
WebDriverWait wait = new WebDriverWait(driver, Duration.ofSeconds(10));

WebElement loadMoreButton = wait.until(ExpectedConditions.elementToBeClickable(By.cssSelector("button[data-test='mms-search-srp-loadmore']")));
loadMoreButton.click();
driver.manage().timeouts().implicitlyWait(Duration.ofMillis(1000));
// stuff to do later ...
driver.quit();
    }

Here is the error message I got in intellij:

Juli 22, 2023 5:03:23 PM org.openqa.selenium.devtools.CdpVersionFinder findNearestMatch
WARNING: Unable to find an exact match for CDP version 115, so returning the closest version found: 114
Exception in thread "main" org.openqa.selenium.TimeoutException: Expected condition failed: waiting for element to be clickable: By.cssSelector: button[data-test='mms-search-srp-loadmore'] (tried for 10 second(s) with 500 milliseconds interval)

I am very thankful for any help!!!

I want to web scrape all computers there and collect all the data, but the problem is there is the load more button. So it just scrapes about 10 computers and that's it. But I want to scrape more computers. So the webdriver should click on the button so it can proceed. To be honest I don't know what the problem is or how it can be fixed. Since I'm pretty new to java and selenium webdriver I don't know where to find a solution, that's why this here is my best shot now, so I hope someone is able to help me out.

Answer 1

First of all, you should close that cookies alert.
Secondly, the Load-More button is not in the user view.
The solution for that is to execute the JS scrollIntoView command before waiting for it to be clickable:

driver.get("https://www.mediamarkt.at/de/category/alle-notebooks-884.html");
driver.manage().timeouts().implicitlyWait(Duration.ofMillis(4000));
WebDriverWait wait = new WebDriverWait(driver, Duration.ofSeconds(10));

// Find the cookie-alert's "close" btn and click on it
By closeAlertLocator = By.xpath("//button[contains(text(),'Speichern')]");
driver.findElement(closeAlertLocator).click();

// Wait for the "close" btn to disappear
wait.until(ExpectedConditions.invisibilityOfElementLocated(closeAlertLocator));

// Locate the "Load More" btn
By loadMoreLocator = By.cssSelector("button[data-test='mms-search-srp-loadmore']");
WebElement loadMoreButton = driver.findElement(loadMoreLocator);

// Scroll to the "Load More" btn and make sure it's in the user view
((JavascriptExecutor) driver).executeScript("arguments[0].scrollIntoView(true);", loadMoreButton);

// Wait for the "Load More" btn to be clickable and click on it
wait.until(ExpectedConditions.elementToBeClickable(loadMoreLocator)).click();

// stuff to do later ...

In addition I've relocated the implicit-wait command. it should be executed before the rest of the code in order to affect it.