Search code examples
linuxiconv

iconv not complete convert to utf8

When I converted my text on this site, be converted correctly:
http://string-functions.com/encodedecode.aspx
I choose source 'Windows-1252' and target 'utf-8'.
See it in the screenshot below:
https://i.sstatic.net/2Pn4E.png

But when I convert with the following code, Some letters are not converted and the text is disrupted.

iconv -c -f UTF-8 -t WINDOWS-1252 < mytext.txt > fixed_mytext.txt

A phrase that should be converted:

آموزش Ùˆ نرم اÙزارهای تعمیر مانیتور

If true, the convert should be this phrase:

 آموزش و نرم افزارهای تعمیر مانیتور

plese help me. thank you

Solution

  • The original text was in UTF-8. It got mistakenly interpreted as a text in Windows-1252 and converted from Windows-1252 to UTF-8. This should have never been done. To undo the damage we need to convert the file from UTF-8 to Windows-1252, and then just treat it as a UTF-8 file.

    There's a problem however. The letter ف is encoded in UTF-8 as 0xd9 0x81, and the code 0x81 is not a part of Windows1252.

    Luckily when the first erroneous conversion was made, the character was not lost or replaced with a question mark. It got converted to a control character 0xc2 0x81.

    The 0xd9 code is in Windows1252, it's the letter Ù, which in UTF-8 is 0xc3 0x99. So the final byte sequence for ف in the converted file is 0xc3 0x99 0xc2 0x81.

    We can just replace with something ASCII-friendly with a sed script, make an inverse conversion, and then replace it back with ف.

    LANG=C sed $'s/\xc3\x99\xc2\x81/===FE===/g' forum.txt  | \
       iconv -f utf8 -t cp1252 | \
       sed $'s/===FE===/\xd9\x81/g'

    The result is the original file encoded in UTF-8.

    (make sure that ===FE=== is not used in the text first!)