Search code examples
phpweb-scrapingsimple-html-dom

Scraping data which loaded after scrolling to bottom

I'm working with PHP and I want to scrape some data from any website. But I have a problem. I scrape data but these items number are 48. But I know that page has 11K items. Rest of datas extend when you scroll and you get new bunch of datas (48 items). I'm scraping with simple_html_dom. How can I manipulate scroll and get data ? Thanks! :)

Solution

  • Sounds like the missing data is loaded via ajax.

    Check the Network tab in the Developer Console (by pressing F12). Take a look at the URL which is being called (and the response), and edit it to your needs. Then call this URL instead of the one you are taking now.