Search code examples
c#asp.netxpathhtml-agility-pack

Html Agility Pack xpath throws null exception


I am trying to parse this page. To select the nodes I need I use XPath, my XPath works fine in my browser, but when using it in my project it returns a null exception. The XPath for title works fine, but the one for description does not.

HtmlWeb web = new HtmlWeb();
HtmlDocument document = web.Load("http://nl.aliexpress.com/item/4000646776468.html?spm=a2g0o.productlist.0.0.531f7aa3iGAnCb&algo_pvid=0b20aa21-fd7f-4826-81a5-c9aac5254da8&algo_expid=0b20aa21-fd7f-4826-81a5-c9aac5254da8-0&btsid=8849a0ec-e95d-447f-a6f9-34dcd58f1381&ws_ab_test=searchweb0_0,searchweb201602_4,searchweb201603_53");
        ProductModel product = new ProductModel { 
             Title = document.DocumentNode.SelectSingleNode("//head/title").InnerText,
             Description = document.DocumentNode.SelectSingleNode("/html/body/div[5]/div/div[3]/div[2]/div[2]/div[1]/div/div[2]/div[1]/div/div/div/div[1]/p[2]").InnerText};
        return View(product);

Solution

  • It indeed turned out to be a problem with content being dynamically rendered. For those who come across the some problem, take a look at selenium if you are using c#. I switched to node using the puppeteer libary.