Search code examples
c#.netwinformswebclient

Use Webclient to get page resource in C#

I use below code for get page Source, but it doesn't return property data :

string url = "http://www.tsetmc.com/Loader.aspx?ParTree=15131F";
            WebClient client = new WebClient();
            client.Headers["Accept-Encoding"] = "gzip";
            string pageSource = client.DownloadString(url);

The Content-Encoding of website is gzip

image

Solution

  • By setting client.Headers["Accept-Encoding"] = "gzip"; you are asking the server to send a compressed response. However, you are not decompressing it. This is causing the incorrect response.

    As per https://stackoverflow.com/a/4914874/23633, you can get WebClient to automatically decompress responses by modifying the HttpWebRequest it creates:

    class MyWebClient : WebClient
{
    protected override WebRequest GetWebRequest(Uri address)
    {
        var request = (HttpWebRequest) base.GetWebRequest(address);
        request.AutomaticDecompression = DecompressionMethods.Deflate | DecompressionMethods.GZip;
        return request;
    }
}

string url = "http://www.tsetmc.com/Loader.aspx?ParTree=15131F";
WebClient client = new MyWebClient();
// don't set the Accept-Encoding header here; it will be done automatically
string pageSource = client.DownloadString(url);