Search code examples
c#html-agility-packweb-crawlergoogle-local-search

speed up parsing in html agility pack


This is a method I use to grab certain tags with the html agility pack. I use this method to do rankings with google local. It seems to take quite a bit of time and be memory intensive, does anyone have any suggestions to make it better?

 private void findGoogleLocal(HtmlNode node) {

     String   name        = String.Empty;
     // 
     // ----------------------------------------
     if (node.Attributes["id"] != null) {

       if (node.Attributes["id"].Value.ToString().Contains("panel_") &&   node.Attributes["id"].Value.ToString() != "panel__")
        {
        GoogleLocalResults.Add(new Result(URLGoogleLocal, Listing, node, SearchEngine.Google, SearchType.Local, ResultType.GooglePlaces));
        }
    }

    if (node.HasChildNodes) {
      foreach (HtmlNode children in node.ChildNodes)  {
        findGoogleLocal(children);
      }
    }

  }

Solution

  • Why does this method have to be recursive? Just get all the nodes in one go (example using the Linq support in HAP):

    var results = node.Descendants()
                      .Where(x=> x.Attributes["id"]!= null && 
                                 x.Attributes["id"].Value.Contains("panel_") &&  
                                 x.Attributes["id"].Value!= "panel__")
                      .Select( x=> new Result(URLGoogleLocal, Listing, x, SearchEngine.Google, SearchType.Local, ResultType.GooglePlaces));