Search code examples
pythonasp.net.netscrapyviewstate

how to scrape a page request using Viewstate parameter?

I've been trying to scrape this website : http://www.e3050.com/Cases-Fans-PDU/C

I can scrape anything normally but moving to the next page, after debugging I found that they send the __Viewstate parameter for each page request. The viewstate parameter is stored in each page response, so I figured out that I need to get it per page and send it to the following page. I get the __viewstate using this xpath :

sel.xpath('//input[@id="__VIEWSTATE"]/@value').extract()

I also got an error, because the viewstate they send is different than the one enlisted in their page response, both parameters are 64 base encoded but the one they send per request has more data than the one I get from the page response.

how can I deal with this ? and how does they get the viewstate parameter ?

Edit: The same issue for __EVENTVALIDATION parameter.

Solution

  • __VIEWSTATE is a parameter used in .net for a security reason. it's basically a hash of the referral page to the page you're requesting. it's usually embedded in an tag, you just need to extract it before each request and add it to your next request.