pySerial decode function returns error 'utf-8 codec can't decode byte 0xff ...'

Question

I set up an Arduino to give this kind of input:

3.30x530.00x2.59x1325.0x28.70x1013.14x0.91x23.41
3.50x531.00x2.60x1327.5x28.70x1013.11x1.17x23.41
3.60x531.00x2.60x1327.5x28.70x1013.14x0.89x23.41

I want the numbers to be send to Python to acess and process them. It works for one or two lines until this error appears 'utf-8' codec can't decode byte 0xff in position 1: invalid start byte

The problem is related to this line

arduino_data = arduino_data[0:len(arduino_data)].decode("utf-8")

This is the complete function:

import time
import schedule
import pandas as pd


def main_func():
    arduino = serial.Serial('COM3', 9600)
    print('Established serial connection to Arduino')
    arduino_data = arduino.readline()
    arduino_data = arduino_data[0:len(arduino_data)].decode("utf-8")

    decoded_values = str(arduino_data)
    list_values = decoded_values.split('x')

    for item in list_values:
        list_in_floats.append(float(item))

    print(f'Collected readings from Arduino: {list_in_floats}')


    arduino_data = 0
    list_in_floats.clear()
    list_values.clear()
    arduino.close()
    print('Connection closed')

I tried changing the arduino code to give a different data type. neither float nor int worked so I started looking into the documentation and the pyhton script but i found nothing so far. what is strange is that it works a couple of times and den suddendly doesnt.

Answer 1

So I tried different encodings (utf-8, cp1252, iso-8859-1, latin1, utf-16) and it turned out that this did not solve my problem but using 'latin1' helped me find out that one of my sensors gave weird redings sometimes. I added a try-loop to filter them and now my code is running even if some errors occur

def main_func():
    arduino_data = arduino.readline()
    dec  = 'latin1' # Möglichkeiten: utf-8, cp1252, iso-8859-1, latin1, utf-16

try:
    arduino_data2 = arduino_data[0:len(arduino_data)].decode(dec)
    decoded_values = str(arduino_data2)
    list_values = decoded_values.split('x') # am x werden die Daten getrennt
    timestamp = str(time.time())

    for item in list_values:
        list_in_floats.append(float(item)) # die augeteilten Daten in eine Liste setzen
    list_in_floats.append(timestamp)
    df.loc[len(df.index)] = list_in_floats # Daten in DataFrame speichern

    print(f'{list_in_floats}')
    # print(list_in_floats[0]) # druckt piezo_spannung

except:
   print("An exception occurred", arduino_data[0:len(arduino_data)].decode(dec))


arduino_data = 0
list_in_floats.clear()
list_values.clear()