Arranging real-time sensor input data based on timestamp

Question

I have real-time time series data from two different sensor input. I would like to arrange the data according to the timestamp.

The problem is that the sensor input is recieved inconsistent, e.g.

 sensor      time  data1     data2     data3     data 4  
['sensor_1', '0', '0.1234', '0.3251', '1.4235', '0.4234']
['sensor_1', '1', '0.1432', '0.3452', '1.4245', '0.1434']
['sensor_2', '0', '0.6543', '0.3231', '1.1235', '0.3434']
['sensor_1', '2', '0.1654', '0.3243', '1.5445', '0.1234']
['sensor_2', '1', '0.1544', '0.3345', '1.5425', '0.6534']
['sensor_2', '2', '0.5432', '0.4551', '1.8755', '0.4245']

How can I join them (while I recieve data) so sensor data with same timestamp are combined, like so:

 time  data1_1   data1_2   data1_3   data1_4   data2_1   data2_2   data2_3   data2_4    
['0', '0.1234', '0.3251', '1.4235', '0.4234', '0.6543', '0.3231', '1.1235', '0.3434']
['1', '0.1432', '0.3452', '1.4245', '0.1434', '0.1544', '0.3345', '1.5425', '0.6534']
['2', '0.1654', '0.3243', '1.5445', '0.1234', '0.5432', '0.4551', '1.8755', '0.4245']

I recieve data through a subprocess (one line at a time):

process = subprocess.Popen(['node', 'server.js'], stdout=subprocess.PIPE)
while True:
     Byte_data = process.stdout.readline()
     string_data = byte_data.rstrip().decode("utf-8") # From byte to string
     list_data = string_data.split(",") # Comma separate strings

     # Sort data here...

Answer 1

You will need to keep a look back and match timestamps as they come in, something like this.

• From the formulation of your question, it seems the timestamps are matching. If they are not, but only ordered, things get a bit more complicated.
• If some frames are missing, they will linger indefinitely in the lookback dicts. If this thing is running for long periods of time, you might want to clear out old frames regularly.

def time(row):
    return row[1]

def synchronized(rows):
    lookback = {'sensor_1': {}, 'sensor_2': {}}
    for row in rows:
        # add row to lookback
        sensor = row[0]
        lookback[sensor][time(row)] = row

        # check for matching timestamps
        rms = []
        for t, r in sorted(lookback['sensor_1'].items(), key=time):
            if t in lookback['sensor_2']:
                yield r, lookback['sensor_2']
                rms.append(t)

        for rm in rms:
            for tmp in lookback.values():
                del tmp[rm]

def parse(rows):
    for row in rows:
        yield tuple(row.decode().rstrip().split(','))

import subprocess
process = subprocess.Popen(['node', 'server.js'], stdout=subprocess.PIPE)
for row1, row2 in synchronized(parse(process.stdout)):
    print(row1, row2)