How do I decode an encoded HttpWebResponse?

Question

I have this piece of code to fetch a Page HTML from an URL, however the response content looks encoded.

Code:

    HttpWebRequest xhr = (HttpWebRequest) WebRequest.Create(new Uri("https://www.youtube.com/watch?v=_Ewh75YGIGQ"));
        xhr.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;
        //xhr.CookieContainer = request.Account.CookieContainer;
        xhr.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
        xhr.Headers["Accept-Encoding"] = "gzip, deflate, br";
        xhr.Headers["Accept-Language"] = "en-US,en;q=0.5";
        xhr.Headers["Upgrade-Insecure-Requests"] = "1";
        xhr.KeepAlive = true;
        xhr.UserAgent = "Mozilla/4.0 (compatible; MSIE 9.0; Windows NT 6.1)";
        xhr.Host = "www.youtube.com";
        xhr.Referer = "https://www.youtube.com/watch?v=6aCpYxzRkf4";
        var response = xhr.GetResponse();
        string html;
        using (StreamReader reader = new StreamReader(response.GetResponseStream()))
        {
            html = reader.ReadToEnd();
        }

These are the response headers:

    X-XSS-Protection: 1; mode=block; report=https://www.google.com/appserve/security-bugs/log/youtube
    X-Content-Type-Options: nosniff
    X-Frame-Options: SAMEORIGIN
    Strict-Transport-Security: max-age=31536000
    Content-Encoding: br
    Transfer-Encoding: chunked
    Alt-Svc: quic=":443"; ma=2592000; v="44,43,39,35"
    Cache-Control: no-cache
    Content-Type: text/html; charset=utf-8
    Date: Sat, 24 Nov 2018 11:30:38 GMT
    Expires: Tue, 27 Apr 1971 19:44:06 EST
    P3P: CP="This is not a P3P policy! See http://support.google.com/accounts/answer/151657?hl=it for more info."
    Set-Cookie: PREF=f1=50000000&al=it; path=/; domain=.youtube.com; expires=Thu, 25-Jul-2019 23:23:38 GMT
    Server: YouTube Frontend Proxy

And the response string parsed with StreamReader.ReadToEnd() looks like this

Answer 1

The answer is in the response header: Content-Encoding: br -> This means Brotli compression.

There is a .NET implementation (NuGet package) for it:

Install this to your project add "using Brotli; " and replace the "using (StreamReader....." with this code:

       using (BrotliStream bs = new BrotliStream(response.GetResponseStream(), System.IO.Compression.CompressionMode.Decompress)) {
            using (System.IO.MemoryStream msOutput = new System.IO.MemoryStream()) {
                bs.CopyTo(msOutput);
                msOutput.Seek(0, System.IO.SeekOrigin.Begin);
                using (StreamReader reader = new StreamReader(msOutput)) {
                    html = reader.ReadToEnd();
                }
            }
        }