I am trying to parse this page. To select the nodes I need I use XPath, my XPath works fine in my browser, but when using it in my project it returns a null exception. The XPath for title works fine, but the one for description does not.
HtmlWeb web = new HtmlWeb();
HtmlDocument document = web.Load("http://nl.aliexpress.com/item/4000646776468.html?spm=a2g0o.productlist.0.0.531f7aa3iGAnCb&algo_pvid=0b20aa21-fd7f-4826-81a5-c9aac5254da8&algo_expid=0b20aa21-fd7f-4826-81a5-c9aac5254da8-0&btsid=8849a0ec-e95d-447f-a6f9-34dcd58f1381&ws_ab_test=searchweb0_0,searchweb201602_4,searchweb201603_53");
ProductModel product = new ProductModel {
Title = document.DocumentNode.SelectSingleNode("//head/title").InnerText,
Description = document.DocumentNode.SelectSingleNode("/html/body/div[5]/div/div[3]/div[2]/div[2]/div[1]/div/div[2]/div[1]/div/div/div/div[1]/p[2]").InnerText};
return View(product);
It indeed turned out to be a problem with content being dynamically rendered. For those who come across the some problem, take a look at selenium if you are using c#. I switched to node using the puppeteer libary.