Search code examples
c#linqxpathhtml-agility-pack

Use XPath to find all JavaScript type script elements


I am trying to find all <script> elements in a series of HTML-documents by using html-agility-pack (plus LINQ and XPath). The documents have script-elements placed in header, and a Google Analytics in footer. First I am trying to target the header scripts and remove them. My Notepad++ shows me that I have 719 script-elements present, but my console appliaction only finds 55 of them.

I need som help to target them properly, so I can remove them from the document.

Source document (structure of head),

<!doctype html system "html.dtd">
<html>
<head>
<link rel="stylesheet" href="../IRstyle.css" type="text/css">
<title>Non-hierarchic document clustering</title>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<meta name="keywords" content="">
<meta name="VW96.objecttype" content="Document">

<script language="JavaScript" type="text/JavaScript">
//Javascript-code goes here
</script>
</head>
<body>    
<!--Body contents goes here-->

<!-- in footer -->
<script src="http://www.google-analytics.com/urchin.js" type="text/javascript">
</script>
<script type="text/javascript">
_uacct = "UA-67XXXX-X";
urchinTracker();
</script>
</body>
</html>

So far I've tried targeting the 'Language' type with JavaScript, but only get a few hits when parsing html/head. My method takes filenames from a list. For now, method prints out number of scripts collected in list, this will change to 'Scripts.Remove();' once I get the search string right.

private static void FindTagsToRemove(IEnumerable<string> files)
{
    var doc = new HtmlDocument();
    List<string> scripts = new List<string>();
    List<string> errors = new List<string>();
    try
    {
        foreach (var file in files)
        {
            doc.Load(@file);
            var head = doc.DocumentNode.SelectSingleNode("html/head");
            var nodes = new List<HtmlNode>();
            bool isScript = false;

            foreach (var node in head.ChildNodes.ToList())
            {
                if (node.NodeType == HtmlNodeType.Element && node.Name.Contains("script"))
                {
                    isScript = !isScript;
                    scripts.Add(node.OuterHtml);
                    Console.WriteLine(node.OuterHtml);
                }
                else if (isScript)
                {
                    nodes.Add(node);
                    node.Remove();
                }
            }
        }
        int nr_scripts = scripts.Count();
        Console.WriteLine("Number of scripts in collection: {0}", nr_scripts);
    }
    catch (Exception Ex)
    {
        Console.WriteLine(Ex.Message);
    }
}

If someone have a better way of targeting the JavaScripts in the head-node, it will be much appreciated. ANY help is appreciated! :)


Solution

  • If you only need <script> element nodes, use descendant-or-self (//). Sample HTML:

    var html =
    @"<!doctype html system 'html.dtd'>
    <html>
    <head>
    <link rel='stylesheet' href='../IRstyle.css' type='text/css'>
    <title>Non-hierarchic document clustering</title>
    <meta http-equiv='Content-Type' content='text/html; charset=iso-8859-1'>
    <meta name='keywords' content=''>
    <meta name='VW96.objecttype' content='Document'>
    <script language='JavaScript' type='text/JavaScript'>
    //Javascript-code goes here
    </script>
    </head>
    <body>    
    <!--Body contents goes here-->
    
    <!-- in footer -->
    <script src='http://www.google-analytics.com/urchin.js' type='text/javascript'>
    </script>
    <script type='text/javascript'>
    _uacct = 'UA-67XXXX-X';
    urchinTracker();
    </script>
    </body>
    </html>";
    

    Parse sample:

    var document = new HtmlDocument();
    document.LoadHtml(html);
    // target only <script> in <head>
    // var scriptTags = document.DocumentNode.SelectNodes("//head/script");
    var scriptTags = document.DocumentNode.SelectNodes("//script");
    
    foreach (var script in scriptTags) script.Remove();    
    
    document.Save(OUTPUT);
    

    Output:

    <!doctype html system 'html.dtd'>
    <html>
    <head>
    <link rel='stylesheet' href='../IRstyle.css' type='text/css'>
    <title>Non-hierarchic document clustering</title>
    <meta http-equiv='Content-Type' content='text/html; charset=iso-8859-1'>
    <meta name='keywords' content=''>
    <meta name='VW96.objecttype' content='Document'>
    
    </head>
    <body>    
    <!--Body contents goes here-->
    
    <!-- in footer -->
    
    
    </body>
    </html>