Search code examples
pythonlinuxfifo

New to fifo in linux, reading with python produces gibberish? (binary)

I am trying to grab some sensor data so I can toss it into a influxdb. We have very old sensors writing this out to a server via a fifo on that server. I know I can read it (made a quick program):

# reader.py
import os

FIFO = '/var/axis/sensor1'

FIFO_PATH = FIFO

fifo_path = FIFO
print("about to "+fifo_path)
# Ensure the FIFO exists
if not os.path.exists(fifo_path):
    raise IOError("The FIFO '{0}' does not exist. Please create it using 'mkfifo {0}'.".format(fifo_path))


print("Opening FIFO {0} for reading...".format(fifo_path))
fifo_fd = os.open(fifo_path, os.O_RDONLY)
fifo_file = os.fdopen(fifo_fd)

try:
  while True:
      line = fifo_file.read(1)
      if not line:
        break

    # Print the line to the screen
      print(line),

finally:
  fifo_file.close()
  print("FIFO Closed.")

So when I run this and the sensor is having data ran to I see stuff like this scroll past the screen:

@eN?>??Y?n??A?)??[Z@?c
@eN?>??\?n??A?)??[Z@?(
@eN?>`?n??A?)??[Z@??    @eN?>33c?n??A?)??[Z@??  @eN?>fff?n??A?)??[Z@??  @eN?>??i?n??A?)??[Z@?   @eN?>??l?n??A?)??[Z@?   @eN?>p?n??A?)??[Z@? @eN?>33s?n??A?)??[Z@?   @eN?>ffv?n??A?)??[Z@?   @eN?>??y?n??A?)??[Z@?   @eN?>??|?n??A?)??[Z@?   @eN?>??n??A?)??[Z@? @eN?>33??n??A?)??[Z@?   @eN?>ff??n??A?)??[Z@?   @eN?>????n??A?)??[Z@?   @eN?>?̌?n??A?)??[Z@?    @eN?>??n??A?)??[Z@? @eN?>33??n??A?)??[Z@?   @eN?>ff??n??A?)??[Z@?   @eN?>????n??A?)??[Z@?   @eN?>?̜?n??A?)??[Z@?    @eN?>??n??A?)??[Z@? @eN?>33??n??A?)??[Z@?   @eN?>ff??n??A?)??[Z@?   @eN?>????n??A?)??[Z@?   @eN?>?̬?n??A?)??[Z@?    @eN?>??n??A?)??[Z@? @eN?>33??n??A?)??[Z@?   @eN?>ff??n??A?)??[Z@?   @eN?>????n??A?)??[Z@?   @eN?>?̼?n??A?)??[Z@?    @eN?>??n??A?)??[Z@? @eN?>33??n??A?)??[Z@?   @eN?>ff??n??A?)??[Z@?   @eN?>????n??A?)??[Z@?   @eN?>????n??A?)??[Z@?   @eN?>??n??A?)??[Z@? @eN?>33??n??A?)??[Z@?   @eN?>ff??n??A?)??[Z@?   @eN?>????n??A?)??[Z@?   @eN?>????n??A?)??[Z@?   @eN?>??n??A?)??[Z@? @eN?>33??n??A?)??[Z@@eN?>ff??n??A?)??[Z@?   @eN?>????n??A?)??[Z@?   @eN?>????n??A?)??[Z@?   @eN?>??n??A?)??[Z@? @eN?>33??n??A?)??[Z@?   @eN?>ff??n??A?)??[Z@?   @eN?>????n??A?)??[Z@?   @eN?>????n??A?)??[Z@P?  @???z?>?n??A?)??[Z@?~
@???z?>33?n??A?)??[Z@?_
                       @???z?>ff?n??A?)??[Z@??

Clearly a readline is the wrong command, I guess this data is binary? Are fifo's always binary? If it is binary that means I have to hunt down how this old system/sensor is writing this fifo to know how to then load and parse it?

I want to know what I am reading so I know how to package it as json to send to influxdb.

EDIT: I believe I found some c code of an example of reading the data structure, with the structure can I read this in python?

C Structure of the data:

typedef struct axis_data {
    double now;
    double p;
    double v;
    double current;
    double error;
} AXIS_DATA;

fifoList.fd = open("/var/axis/sensor1", O_RDONLY);
AXIS_DATA data;
rb=read(this->fifoList.fd, &data, sizeof(data));

data.p, data.v, data.current, data.error

Paraphrased C of course, and my C is rusty but I believe I want to do the same thing with creating a structure equiv in Python and reading that structure size into the structure itself. Somehow, in python, my attempt at that below (I cannot believe I have never done binary in python really)

# reader.py
import os
from ctypes import *

class AXIS_DATA(Structure):
    _fields_ = [('now', c_double),
                ('p', c_double),
                ('v', c_double),
                ('current', c_double),
                ('error', c_double)]




FIFO = '/var/axis/sensor1'

#FIFO = 'mytest'
FIFO_PATH = FIFO

fifo_path = FIFO
print("about to "+fifo_path)
# Ensure the FIFO exists
if not os.path.exists(fifo_path):
    raise IOError("The FIFO '{0}' does not exist. Please create it using 'mkfifo {0}'.".format(fifo_path))


print("Opening FIFO {0} for reading...".format(fifo_path))
fifo_fd = os.open(fifo_path, os.O_RDONLY)
#fifo_file = os.fdopen(fifo_fd)
fifo_file = open(fifo_path)

try:
  while True:
      result = []
      dat = AXIS_DATA()
      # this just froze
      while fifo_file.readinto(dat) == sizeof(dat):
        result.append((dat.now, dat.p, dat.v, dat.current, dat.error))
      if not line:
        break

    # Print the line to the screen
      print(result),

finally:
  fifo_file.close()
  print("FIFO Closed.")
Solution

  • "Are fifo's always binary?"

    They are agnostic to the data that they "carry". Just like files, sockets, etc.

    • If you send binary data in one end you get the same binary data out of the other end.
    • If you send text data with a specific encoding (ASCII, ISO-Latin, UTF-8, ...), you get back text with the same encoding.

    The gibberish you are seeing looks like binary data that has been (mistakenly) read as UTF-8 encoded text. (The ? characters are most likely substitutions for bytes / byte sequences that cannot be translated into printable characters.)

    The approach of treating the data as text is unlikely to work. Each of the ? substitutions is a place where you have lost information. Instead, you will need to reverse engineer the binary format of the sensor's output. (Or maybe the C struct declaration you found is sufficient.)

    Once you have figured out the binary format, you should be able to read it with the help of the built-in Python struct library.

    This module converts between Python values and C structs represented as Python bytes objects. Compact format strings describe the intended conversions to/from Python values. The module’s functions and objects can be used for two largely distinct applications, data exchange with external sources (files or network connections), or data transfer between the Python application and the C layer.

    (The ctypes library is for a different purpose: translating data types when making function calls to external "native code" libraries / DLLs. It won't help here.)