encoding as windows-1252 and decoding as UTF-8

Question

recently I've stumbled upon this old python code:

for key, value in values.items():
    item = value
    try:
        if type(item) is str:
            item = item.encode('windows-1252')
            item = item.decode('utf8')
    except BaseException:
        item = value
    finally:
        parsed_values[key] = item
return parsed_values

Is there a valid reason to do such encoding conversion? Isnt utf8 going to not show some of windows-1252 characters if they are present?

Answer 1

Is there a valid reason to do such encoding conversion?

Yes, of course - if the data in the values had been incorrectly decoded as windows-1252 in the first place, this is the correct thing to do.

The correct thing to do would be to not have this data decoded to windows-1252 wherever it goto read into the process. (the data must have come from someplace: a file, a network connection, a database query) - whatever the source, there is an encoding name associated to the call - which is used to tranform the incoming bytes to an str object. utf-8 should be set at this point.

utf-8 is used by default everywhere but on Windows file reading. The problem of having the code as above baked in the system is that if (or rather when) the same app is run in another platform, this will break the text instead of fixing it.

All that said:

Isn't utf8 going to not show some of windows-1252 characters if they are present?

To the contrary windows-1252 can only represent 256 characters, while UTF-8 can represent any of the millions of Unicode characters possible, even future ones.

---

Here is what is going on here:

The data the app is dealing with is coded as utf-8 on disk, for example with the following byte sequece:

In [3]: b                                                                                
Out[3]: b'ma\xc3\xa7\xc3\xa3'
# then, this is read from a file oppened with the incorrect encoding information - which is equivalent to do:
In [4]: c = b.decode("windows-1252")                                                     

In [5]: c                                                                                
Out[5]: 'maçã'
# the code above translate the codepoints between 128 and 255 in a "raw"
# form back to bytes - and then correctly decode then as utf-8:
In [6]: d = c.encode("windows-1252").decode("utf-8")                                     

In [7]: d                                                                                
Out[7]: 'maçã'