UnicodeDecodeError in Python3 when reading binary files stdin

Question

I'm trying to take what's in the stdin and read it 1022 bytes at a time. This code runs fine for text files. But when inputting a binary file it gives me the UnicodeDecodeError. Where data below is sys.stdin.

def sendStdIn(conn, cipher, data):
    while True:
        chunk = data.read(1022)
        if len(chunk)==1022:
            EOFAndChunk = b'F' + chunk.encode("utf-8")
            conn.send(encryptAndPad(cipher,EOFAndChunk))
        else:
            EOFAndChunk = b'T' + chunk.encode("utf-8")
            conn.send(encryptAndPad(cipher,EOFAndChunk))
            break
    return True

The binary file was made by calling dd if=/dev/urandom bs=1K iflag=fullblock count=1K > 1MB.bin

I run the file with essentially python A3C.py < 1MB.bin Then I end up with below.

Traceback (most recent call last):
  File "A3C.py", line 163, in <module>
    main()
  File "A3C.py", line 121, in main
    EasyCrypto.sendStdIn(soc, cipher, sys.stdin)
  File "EasyCrypto.py", line 63, in sendStdIn
    chunk = data.read(1022)
  File "/usr/lib64/python3.5/codecs.py", line 321, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe8 in position 0: invalid continuation byte

Any idea how I can make this so it can read sections of the binary file because I need to send them from this client side to the server side piece at a time. Thanks!

Answer 1

sys.stdin is a text wrapper that decodes binary data. Use sys.stdin.buffer instead:

EasyCrypto.sendStdIn(soc, cipher, sys.stdin.buffer)

The TextIOBase.buffer attribute points to the binary buffered I/O object underneath.