I have a large MySQL table filled with Chinese characters in an incorrect encoding. I believe they were supposed to be encoded in latin1 (iso-8859-1), but I just can't find a way to get the Chinese characters from the contents of the database rows.
Converting between latin1 and utf8 doesn't help - the fields remain unchanged. I've tried re-importing the database with various encodings - always the same results.
Some examples of the current contents and what they should be:
惨事 should be 惨事
ä¸ should be 不
最 should be 最
I've also tried using Python to try and 'decode' the contents, but again without success. I've tried various combinations of this:
databasefield.decode('iso-8859-1').encode('utf8')
But I can't get anything like that to work either.
Sorry for asking such a vague question, but I just don't know how to continue trying to figure this out!
Does anyone know what the problem is here?
You are looking at UTF-8 decoded as Windows codepage 1252 instead:
>>> print u'惨事'.encode('utf8').decode('cp1252')
惨事
>>> print u'最'.encode('utf8').decode('cp1252')
最
Fixing this requires going the other way:
>>> print u'惨事'.encode('cp1252').decode('utf8')
惨事
>>> print u'最'.encode('cp1252').decode('utf8')
最
There may have been some loss there though, as the UTF-8 encoding for 不 uses a codepoint not supported by 1252:
>>> u'不'.encode('utf8')
'\xe4\xb8\x8d'
>>> print u'不'.encode('utf8').decode('cp1252')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/mj/Development/venvs/stackoverflow-2.7/lib/python2.7/encodings/cp1252.py", line 15, in decode
return codecs.charmap_decode(input,errors,decoding_table)
UnicodeDecodeError: 'charmap' codec can't decode byte 0x8d in position 2: character maps to <undefined>
There are several other Windows codepage candidates that can be tried here though; 1254 would result in similar output, for example, with only minor differences.