awesomium web scraping certain parts

Question

I asked this earlier but I wanted to rephrase the question. I am trying to make a scraper for my project. I would like to have it display a certain part of a link. The only part of the link that changes is the number. This number is what I would like to scrape. The link looks like this:

<a href="/link/player.jsp?user=966354" target="_parent" "="">

As mentioned I am trying to scrap only the 966354 part of the link. I have tried several ways to do this but cant figure it out. When I add

 <a href="/link/player.jsp?user="         

to the code below it breaks

 List<string> player = new List<string>();
 string html = webControl2.ExecuteJavascriptWithResult("document.getElementsByTagName('a')[0].innerHTML");
 MatchCollection m1 = Regex.Matches(html, "<a href=\\s*(.+?)\\s*</a>", RegexOptions.Singleline);
 foreach (Match m in m1)
 {
    string players = m.Groups[1].Value;
    player.Add(players);
 }
    listBox.DataSource = player;

So I removed it, it shows no errors until I go to run the program then I get this error:

"An unhandled exception of type 'System.InvalidOperationException' occurred in Awesomium.Windows.Forms.dll"

So I tried this and it some what works:

string html = webControl2.ExecuteJavascriptWithResult("document.getElementsByTagName('html')[0].innerHTML");    

This code scraps but not the way I would like, Could someone lend a helping hand please.

Answer 1

I would use HtmlAgilityPack (install it via NuGet) and XPath queries to parse HTML.

Something like this:

string html = webControl2.ExecuteJavascriptWithResult("document.getElementsByTagName('html')[0].innerHTML");
var htmlDoc = new HtmlAgilityPack.HtmlDocument();
htmlDoc.LoadHtml(html);

var playerIds = new List<string>();

var playerNodes = htmlDoc.DocumentNode.SelectNodes("//a[contains(@href, '/link/profile-view.jsp?user=')]");

if (playerNodes != null)
{
    foreach (var playerNode in playerNodes)
    {
        string href = playerNode.Attributes["href"].Value;

        var parts = href.Split(new char[] { '=' }, StringSplitOptions.RemoveEmptyEntries);
        if (parts.Length > 1)
        {
            playerIds.Add(parts[1]);
        }
    }

    id.DataSource = playerIds;
}

Also you may find these two simple helper classes useful: https://gist.github.com/AlexP11223/8286153

The first one is extension methods for WebView/WebControl and the second one has some static methods to generate JS code for retrieving elements (JSObject) by XPath + getting coordinates of JSObject)