Search code examples
pythonajaxweb-scraping

Scraping next page XHR request

I want to scrape the second page of this user reviews.

However the next button executes a XHR request, and while I can see it using Chrome developer tools, I cannot replicate it.

Solution

  • It's not so easy task. First of all you should install this extension. It helps you to test own requests based on captured data, i.e. catch and simulate requests with captured data.

    As I see they send a token in this XHR request, so you need to get it in from html page body(stores in source code, js variable "taSecureToken" ).

    Next you need to do four steps:

    1. Catch POST request with plugin
    2. Change token to saved before
    3. Set limit and offset variables in POST request data
    4. Generate request with resulted body

    Note: on this request server returns json data(not the html with next page) containing info about loaded objects on next page.