Search code examples
c#javascriptajaxhttpwebrequest

C# httpwebrequest and javascript

I am using C# HttpWebRequest to get some data of a webpage. The problem is that some of the data is updated using javascript/ajax after the page is loaded and I am not getting it in the response string. Is there a way to have the webrequest wait untill all the scripts in the page have finished executing?

Thanks

Amit

Solution

  • If I correctly interpret your question, there is no simple solution for your problem.

    You are scraping the HTML from a server and since your C# code is not a real web browser, it doesn't execute client scripts.

    This way you can't access information which the HTML you fetch doesn't contain.

    Edit: I don't know how complex these AJAX calls from the original web site are, but you could use Firebug or Fiddler for IE to see how the requests are made in order to call these AJAX calls in your C# application too. So you could add the pieces of information you'll need. But it's only a theoretical solution.