Search code examples
c#webclientencodewindows-1251

C#: WebClient - Cant Unrecognize cyrillic characters


Trying to parse site: link

Code to download content:

WebClient client = new WebClient();
client.Encoding = System.Text.Encoding.ASCII; // OR UTF8
string reply = client.DownloadString(url);

Response:

<!DOCTYPE HTML>
<html prefix="og: http://ogp.me/ns#">
<head><meta http-equiv="Content-Type" content="text/html; charset=windows-1251">
    <link rel="icon" type="image/vnd.microsoft.icon" href="https://spravnik.com/favicon.ico"/>
    <link rel="SHORTCUT ICON" href="https://spravnik.com/favicon.ico"/>
    <link href="/src/main.css?v=1.25" rel="stylesheet" type="text/css" />
    <script src="https://cdn.contentsitesrv.com/js/push/subscribe.js?v=1.3.0"></script>
<title>??????????? 12 ??????? ??. - ?????????? ?????????? ??????</title>
<meta name="keywords" content="?????????? ?????????? ????????????, ???? 09 ????????????, ?????????? ????? ????????????"/>
<meta name="description" content="? ??????????? &#9742; ????????? ?? ??????? ????? ??? ???? ???????????? ?? 12 ??????? ??. ????? ???????? ?? ?????? ????????, ?????? ???????? ????? ???????? ? ????? ?? ?????? ????????."/>
<meta property="og:title" content="?????????? ??????????. ??????????? ? ?? ??????...!"/>

All cyrillic characters are converted in "???" Or in ����


Solution

  • It looks like this site just ignores your Client Encoding and return you data in 1251 encoding. I prefer using RestClient and check responses ContentType. But if you absolutely sure about this site - the code below works correctly.

    WebClient client = new WebClient {Encoding = Encoding.UTF8};
    byte[] reply = client.DownloadData(url);
    
    Encoding.RegisterProvider(CodePagesEncodingProvider.Instance);
    Encoding encoding1251 = Encoding.GetEncoding("windows-1251");
    var convertedBytes = Encoding.Convert(encoding1251, Encoding.UTF8, reply);
    
    string result = Encoding.UTF8.GetString(convertedBytes);