Search code examples
c#ajaxxmlhttprequesthtml-agility-pack

Retrieve AJAX (XHR) infos from a dynamic website in C#

I'm trying to create an app using C# to retrieve my data cap infos from my ISP site. The page is this one but I suspect it can't be accessed from outside their network so if anyone needs more information just ask.

The page loads through AJAX the remaining traffic quota and displays it in the page after it's loaded. Right now I already have a working app using HtmlAgilityPack but it pretty hideous considering I'm using a WebBrowser control to load the page in background, wait for five seconds, parse the page's HTML with the library and see if it finds the necessary HTML string; if not the timer resets and repeats itself until the javascript has done its thing and loaded the data cap infos.

I want to somehow replicate what the webpage does and call the server and ask directly for the infos without creating a web browser instance in background and waiting for the infos to load.

Is it possible?

URL http://internet.tre.it/calls/checkMSISDN.aspx?g=2518607185932962118&h=UItDOr88/CtwONsfqfLgblVuTAysHYKc3kh6mLgiX0He49TU0I9lc56O8mWVhxzd3yFUDFF08P/Ng/5cg2nLtefFfjUIBq/QNQalmmSnKkQ=&mc=22299&acid=0&_=1541582209456

Headers

Host: internet.tre.it
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:63.0) Gecko/20100101 Firefox/63.0
Accept: application/json, text/javascript, */*
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Referer: http://internet.tre.it/
X-Requested-With: XMLHttpRequest
DNT: 1
Connection: keep-alive
Cookie: cookiesAccepted=1; _iub_cs-160673=%7B%22id%22%3A160673%2C%22consent%22%3Atrue%2C%22timestamp%22%3A%222018-04-16T15%3A42%3A10.978Z%22%2C%22version%22%3A%220.13.22%22%7D; ASP.NET_SessionId=n2wz2brfaepfj2klo0nqfwaw; pageVisit=c73074b54dbe40d49a715aeb9a0f4ea8; 148162__148162_d32646f682e342dba303540b0f10dac1=1

Response

Album of the JSON response. (I blacked out those two lines because they were respectively my own name and my phone number)

Solution

  • The response being a json string, I recommend the following :

    • Write code to download the json string from the URL. See for instance https://stackoverflow.com/a/11891101/4180382
    • Copy the whole json string from your F12 response tab
    • In Visual Studio create a new class file
    • Click Edit > Paste special > Paste Json as classes.
    • In your code you will need the name of the first class that you pasted. It is the parent class. I would say it is 'Banners', but verify.
    • var obj = JsonConvert.DeserializeObject < Banners>(downloadedJson);
    • Now you can loop through the Menu array to extract all of the info you need.

    And you are done! If all the info is in the json, no need to use HtmlAgilityPack. Let me know how you fare.