Search code examples
javascriptselenium-webdriverselenium-chromedriver

Html file is missing content when loading with Selenium only after Deploy

I am trying to retrieve the prices from the website https://app.evimdekipsikolog.com/. The prices are within "cards" within the following element

<div class="card-body text-center text-md-left"> 
//html structure
 <b>900₺</b>

Locally I could access them with Selenium by

WebElement card = driver.findElement(By.cssSelector("div[class=\"card-body text-center text-md-left\""));
String price = card.findElement(By.tagName("b")).getText();

But after deploying it to Heroku, the card elements cant be found, because (I assume) they are loaded through a script. The basic structure of the website itself is all loading fine and can be accessed by Selenium.

When printing the html file

System.out.println(driver.getPageSource());

locally, it included all cards, but in Heroku they were not included.

I am now looking for a way to load these Elements, which (I believe) is done through executing the scripts on the page.

I added really long wait elements, hoping the page will load by itself, but nothing changed.

I also tried

JavascriptExecutor js = (JavascriptExecutor) driver;
js.executeScript("document.querySelectorAll('script').forEach(script => eval(script.innerHTML));");

in the hope, that it runs all scripts, but the html file didnt change.

Solution

  • In my particular case ,for some reason the data loaded when entering the landing page with selenum locally, but did not load in deployment.

    However when entering a particular route first, (i.e. https://app.evimdekipsikolog.com/Danismanlar/Busra-Yorulmaz/2239), it triggered the load of all data, and when entering the landing page again, the data was loaded.

    Edit: The reason turned out to be, that heroku hosts its server in the US, and the website had some bugs when accessed outside of turkey