Parsing C# HtmlAgilityPack

Question

My code:

public static (HtmlNodeCollection title, HtmlNodeCollection price) ParsingNodesTP()
{
    HtmlWeb web = new HtmlWeb();
    HtmlDocument doc = web.Load("https://rozetka.com.ua/ua/search/?text=Asus+Zenbook+14&producer=asus&page=1");
    var titles = doc.DocumentNode.SelectNodes("//a[@class='goods-tile__heading ng-star-inserted']//span");
    var price = doc.DocumentNode.SelectNodes("//div[@class='goods-tile__prices']//div[@class='goods-tile__price price--red ng-star-inserted']//p//span[@class='goods-tile__price-value']");
    return (titles, price);
}

The error I got:

System.NullReferenceException: 'Object reference not set to an instance of an object.'

Where is the problem?

Answer 1

F12 is your friend in any browser. Look at the network tab. The data you are interested in is of type "xhr". No need to use HTMLAgilityPack. All you need to do is parse the Json returned by your url.

1. Write code to download the json string from the URL. The following code works for me :

       using System.Net;
   
       using (WebClient wc = new WebClient())
       {             
           wc.Headers.Add("accept", "application/json, text/plain, */*");          
           wc.Headers.Add("user-agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36");              
   
           string downloadedJson = wc.DownloadString("https://search.rozetka.com.ua/ua/search/api/v6/?front-type=xl&country=UA&lang=ua&producer=asus&page=1&text=Asus+Zenbook+14");                 
       }
   

2. Copy the whole result (the value of downloadedJson) on your clipboard.

3. In Visual Studio create a new class file

4. Click Edit > Paste special > Paste Json as classes. In your code you will need the name of the first class that you pasted. It is the parent class called Rootobject by default

5. Install Newtonsoft.Json

 using Newtonsoft.Json
 Rootobject obj = JsonConvert.DeserializeObject < Rootobject>(downloadedJson);

Now you can loop through the Data array to extract all of the job info you need.

        Good[] goods= rootobject.data.goods;