Python cannot decode XORed string with a key larger than 127

Question

I have a problem with decoding bytes to string in python. I wanted to crack the single-byte xor cipher by brute-forcing it. And I did. The key was 90 (dec). The problem here is, that I actually could XOR every byte with a number > 127 but then I always got UnicodeError, when I wanted to decode it. They all looked the same, just different bytes:

128. 'utf-8' codec can't decode byte 0x98 in position 0: invalid start byte

def main():
    d = base64.b64decode(base64string)
    for i in range(1, 255):
        try:
            output_bytes = b''.join([bytes([b ^ i]) for b in d])
            decoded = output_bytes.decode('utf-8')
        except UnicodeDecodeError as e:
            print(f'{i}. {e}')
            pass


if __name__ == '__main__':
    main()

I either had to write a try/except block to prevent my program from failing, or I had to change range(1, 255) to range(1, 127).

Is it an expected behavior? If no, then what am I doing wrong, and if yes, then why? Shouldn't I be able to do xor and then decode() no matter which byte I use?

Answer 1

UTF-8 expects bytes to have a specific format. If you don't have the right sequence of bytes, it can generate a UnicodeDecodeError. Unless you have the correct key, it would be expected to fail. This also assumes the original message was UTF-8-encoded to begin with. Using a try/except is the right thing to do if the message doesn't have the correct key.

For example, here's a Unicode string encoded in UTF-8 and encrypted, then decoded with all possible byte XOR keys. Only a few keys are successful decoding UTF-8:

#coding:utf8
def encode(msg,key):
    data = msg.encode('utf8')
    return bytes(b ^ key for b in data)

def decode(msg,key):
    data = bytes(b ^ key for b in msg)
    try:
        return data.decode('utf8')
    except UnicodeDecodeError:
        return None

msg = "Hello, 马克!"
enc = encode(msg,0x90)
print(enc)

for i in range(256):
    dec = decode(enc,i)
    if dec is not None:
        print(f'{i:02x}: {dec}')

Output:

b'\xd8\xf5\xfc\xfc\xff\xbc\xb0y9<u\x15\x1b\xb1'
90: Hello, 马克!
91: Idmmn-!設䄊 
92: Jgnnm."뫮燉#
93: Kfool/#ꪯ憈"
96: Ncjji*&ﯪネ'
97: Obkkh+'₌&
98: @mddg$(ᡤ퍃)
99: Aleef%)ࠥ쌂(
9a: Boffe&*㣦+
9b: Cnggd'+⢧*
9c: Di``c ,奠鉇-
9d: Ehaab!-䤡舆,
9e: Fkbba".秢닅/
9f: Gjcc`#/榣ꊄ.