I've been parsing this Url with Html Agility Pack:
http://www.cmegroup.com/trading/energy/crude-oil/light-sweet-crude_quotes_settlements_options.html"
The default table displayed is always the closest contract date and the current date.
I have no problem parsing the complete page above but if I ask for another date I can't seem to get a new table when I add a query parameter to get another date:
This still returns the table for the current date. ie. 03/08/2018
However it does work if I add another query for contract month as well:
But if I then query:
....it will not give me the table for 03/06/2018.
It only appears to update the html for me when I change two or more query parameters in the Url. I'm fairly much a Noob with Html so I'm not sure if it's something to do with the actual website 'blocking' my request. Or does it expect some 'user interaction'?
The very 'basic' core of my code is:
using HtmlAgilityPack;
HtmlDocument htmlDoc = new HtmlDocument { OptionFixNestedTags = true };
HtmlWeb web = new HtmlWeb();
htmlDoc = web.Load(url);
A step in the right direction would be great.
Thank you.
Its a ajax site. the WepPage contains JS that makes Ajax queries by filtering done.
Therefore you do not need the html-agility-pack but the JSON.NET.
you need build the url query string, download th text with WebClient.DownloadString and convert it to POCO with JSON.NET.