Search code examples
c#htmlgoogle-chromehttpwebrequest

Why HttpWebRequest returns significantly different html source comparing to chrome > view page source?

I need to get HTML source code as possible as near to normal page view source of chrome or other browser. But following code returns different code for same URL.

String url = @"https://m.facebook.com";
try
{
   HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
   HttpWebResponse response = (HttpWebResponse)request.GetResponse();
   if (response.StatusCode == HttpStatusCode.OK)
   {
      Stream receiveStream = response.GetResponseStream();
      StreamReader readStream = null;
      if (response.CharacterSet == null)
         readStream = new StreamReader(receiveStream);
      else
         readStream = new StreamReader(receiveStream,
                     Encoding.GetEncoding(response.CharacterSet.Replace("\"", string.Empty)));
      //readStream = new StreamReader(receiveStream, Encoding.GetEncoding(response.CharacterSet));
      string data = readStream.ReadToEnd();
      response.Close();
      readStream.Close();
      //string[] sps = data.Split(new string[] { @"videoId"":""" }, StringSplitOptions.RemoveEmptyEntries);
   }
}
catch (Exception ex)
{}

It returns following:

<!DOCTYPE html>
<html lang="en" id="facebook" class="no_js">
<head><meta charset="utf-8" /><meta name="referrer" content="default" id="meta_referrer" /><script nonce="EW2LyNr7">window._cstart=+new Date();</script><script nonce="EW2LyNr7">function envFlush(a){function b(b){for(var c in a)b[c]=a[c]}window.requireLazy?window.requireLazy(["Env"],b):(window.Env=window.Env||{},b(window.Env))}envFlush...

But browser source is..

<!DOCTYPE html><html><head><script id="u_0_2" nonce="db1veTby">"use strict";window.MPageLoadClientMetrics=function(){var a=+new Date(),b={prelude_onload:["jewels_visible","first_paint","visibility_change","tti"],nav_started:["first_paint","visibility_change","prelude_onload"],first_paint:["jewels_visible","visibility_change","prelude_onload"],jewels_visible:["tti","visibility_change","navigation","prelude_onload"],tti:["e2e","visibility_change","navigation"]},c=3,d=3,e="nav_started",f=!0,g="",h="",i=1,j="",k="",l="",m=function(){},n=!0,o=!1,p=!1,q=[],r=window.performance||window.msPerformance||window.webkitPerformance||{},s=(window.requestAnimationFrame||window.webkitRequestAnimationFrame||window.mozRequestAnimationFrame||window.oRequestAnimationFrame||window.msRequestAnimationFrame||window.setTimeout).bind(window),t=window.location.origin||window.location.protocol+"//"+window.location.hostname+(window.location.port&&":"+window.location.port);function u(b,c,d,e,f,i){r.timing&&r.timing.navigationStart&&(a=r.timing.navigationStart),j=b,k=c,l=d,g=e,h=f,n=i,x()}function v(a){var c=b[e];return c&&c.indexOf(a)!==-1}function w(a){return!b[a]}function x(){var a,b;do

How can I get the similar code to view page source of chrome?

Solution

  • Facebook require User-Agent to display the page properly, otherwise it redirects to /unsupportedbrowser page.

    Here's example using HttpClient

    class Program
{
    private static readonly HttpClient client = new HttpClient();

    static async Task Main(string[] args)
    {
        client.DefaultRequestHeaders.UserAgent.ParseAdd("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.105 Safari/537.36");

        string result = await client.GetStringAsync("https://m.facebook.com");
        Console.WriteLine(result);
        Console.ReadKey();
    }
}

    The output is exactly the same as in Google Chrome.