Search code examples
pythonpandasindexingdata-storage

How to split 1 line of data into multiple columns with pandas


I am currently working with CAN-Bus data that outputs everything into 1 column, multiple rows. The way the code works is by sending and receiving msg to a car's Can-bus network to receive real-time data. The received data is stored in a variable "response" which when printed out, gives the following data.

Timestamp: 1646874440.563719        ID: 0060    S Rx                DL:  8    00 00 00 00 00 00 00 08     Channel: can0
Timestamp: 1646874440.563959        ID: 0090    S Rx                DL:  8    00 00 c2 00 61 83 00 00     Channel: can0
Timestamp: 1646874440.564195        ID: 00c0    S Rx                DL:  8    00 2d 61 27 50 00 39 f3     Channel: can0
Timestamp: 1646874440.565563        ID: 0010    S Rx                DL:  8    0d e7 00 00 00 00 81 47     Channel: can0
Timestamp: 1646874440.566430        ID: 00c8    S Rx                DL:  8    cb 87 3e f1 10 38 c6 02     Channel: can0
Timestamp: 1646874440.567062        ID: 0160    S Rx                DL:  8    55 ff 00 00 57 23 ed ff     Channel: can0
Timestamp: 1646874440.567308        ID: 0180    S Rx                DL:  8    03 00 0d f8 17 fe f0 00     Channel: can0
Timestamp: 1646874440.567558        ID: 0190    S Rx                DL:  8    80 00 80 00 80 00 80 00     Channel: can0
Timestamp: 1646874440.567802        ID: 01c0    S Rx                DL:  8    f7 c0 00 00 80 00 00 00     Channel: can0
Timestamp: 1646874440.568048        ID: 01d0    S Rx                DL:  8    a8 ff ff 00 00 00 00 00     Channel: can0
Timestamp: 1646874440.568282        ID: 01e0    S Rx                DL:  8    58 e1 de 05 00 00 fd 23     Channel: can0
Timestamp: 1646874440.568531        ID: 0210    S Rx                DL:  8    00 00 00 00 f2 9b 00 00     Channel: can0
Timestamp: 1646874440.568772        ID: 00a0    S Rx                DL:  8    01 7b 29 f3 00 00 7f fe     Channel: can0
Timestamp: 1646874440.569017        ID: 0213    S Rx                DL:  8    00 f2 00 00 0f ff df ff     Channel: can0

I am trying to look for a way to separate this data into separate "Timestamp" "ID" "DL" "Channel" columns and separate ones for each hex value. I need them separated in order to sort through and get the codes and values needed for different display data (Ex. RPM, Steering angle). This data is set to update every 1/10 of a second however it can increase, I don't know if a system should be put in place to deleted old data after a certain amount of time so it doesn't fill up the data storage.

I am rather new to python indices and data logging so anything would help.

Thank you for your time!

Code I am currently testing with (Clarifications in the comments).

import pandas as pd

df = pd.read_table("CB.txt" , delimiter=" ")
df = df.str.split(" ", 1, expand = True)
print(df)

Solution

  • OK, so I did end up finding a solution. The biggest problem is that the white spaces count as an element in a table so you have to make a column for the white space and deleted the column right after.

    The following code worked for me.

    import pandas as pd
    
    df = pd.read_table("CB.txt", sep=" ", header=None)
    df.columns = ["a","Timestamp","c","d","e","f","g","h","i","j","ID","l","m","n","o","p","q","r","s","t","u","v","w","x","y","z","1","2","3","4","5","6","7","fag","9","10","11","B0","B1","B2","B3","B4","B5","B6","B7","20","21","22","23","24","Channel"]
    df = df.drop(columns=['a','c','d','e','f','g','h','i','j','l','m','n','o','p','q','r','s','t','u','v','w','x','y','z','1','2','3','4','5','6','fag','7','9','10','11','20','21','22','23','24'])   
    

    An error will appear stating the number of elements needed, so this list will change according to what you need. This is definitely not the best way to do this, but its a solution non the less.