Selenium webscraping issue, data not loading

Question

I have an issue, I try scraping a website, what I'm scraping is a table and I get every element, row by row.

Where I have an issue is that an element seems to be loaded only if you scroll to it and let it stay in the field of view for a little while.

Here is the code I use to scrape the text of the element :

tspan_element = row.find_element(By.CSS_SELECTOR, "text > tspan")
row_data.append(tspan_element.text)

I would like some help to find a solution to load all of the element, because I manage to get the content of the first 2 row but not the rest. If you want the website : https://ereferer.com And the page I want to scrap : https://ereferer.com/bo/exchange-site-find

Answer 1

The solution to solve the issue of element not loading if not in view is to simulate a scroll, I did that with a script.

scroll_script = """
        const scrollHeight = document.documentElement.scrollHeight;
        const windowHeight = window.innerHeight;
        const scrollStep = Math.floor(scrollHeight / 100);
        let scrollPosition = 0;
        function scrollToBottom() {
            if (scrollPosition < scrollHeight) {
                scrollPosition += scrollStep;
                window.scrollTo(0, scrollPosition);
                setTimeout(scrollToBottom, 50);
            }
        }
        scrollToBottom();
        """

Then I did a driver.execute_script(scroll_script) to execute the script. It will make you scroll slowly to the bottom of the page so every element have the time to generate itself.