np.fromfile returns blank ndarray

I'm supposed to troubleshoot an open-source Python 2.7 code from GitHub that for some reason, cannot import the binary file correctly. I traced the problem to the following definition, which supposedly imports a binary file (.dat) into an ndarray (rawsnippet) with the following code

    def update_rawsnippet(self):

        self.rawfile_samp = self.rawfile.tell()/self.datatype.itemsize
        self.rawfile_time = self.rawfile_samp/self.fs

        rawsnippet = np.fromfile(self.rawfile, self.datatype, self.S)
        self.rawsnippet = self.format_rawsnippet(rawsnippet)
        return

where self.rawfile is an open file object defined in the following

    def open_rawfile(self):

        if (not hasattr(self, 'rawfile')) or (hasattr(self, 'rawfile') and self.rawfile.closed):
            self.rawfile = open(self.abspath, 'rb')
            print('Opened rawfile = \'%s\'.\n'%self.abspath)
        else:
            print('Rawfile = \'%s\' is already opened.\n'%self.rawfile.name)
        return

and self.datatype and self.S are

self.datatype=np.dtype([('i', np.int16), ('q', np.int16)])
self.S = 2500

But for some reason, rawsnippet is just an empty array. When I print rawsnippet, it simply returns [].

I've tried testing the function np.fromfile and checked the binary file with the following code, and it works.

>>>rawfile=open(abspath,'rb')
>>>rawsnippet = np.fromfile(rawfile,datatype,s)
>>>print rawsnippet
[( 1280,     0) ( 1280,     0) ( 1280,     0) ... (-1537,  1280)
 ( -769, -1025) ( -769, -1025)]

So really don't know why it don't work in the full code.

Interestingly, when I change the datatype to the following (in the full code)

datatype = np.dtype([('i', np.int8), ('q', np.int8)])

rawsnippet is no longer empty, but reads the binary file wrongly (The values are wrong).

Any clues as to why this occurs?? I'm using numpy ver 1.16.6

Note:

The full code can be found at the following link.

https://github.com/mjp5578/GGRG-GPS-SDR

I'm trying to run the PyGNSS code, and def update_rawsnippet(self) as well as def open_rawfile(self) can be found in rawfile.py

Solution

In [225]: datatype=np.dtype([('i', np.int16), ('q', np.int16)])
     ...: S = 2500

Make an empty file:

In [226]: !touch empty.txt

Load with this, or any dtype, produces an (0,) shape file:

In [227]: np.fromfile('empty.txt', dtype=datatype)
Out[227]: array([], dtype=[('i', '<i2'), ('q', '<i2')])
In [228]: np.fromfile('empty.txt', dtype=datatype, count=S)
Out[228]: array([], dtype=[('i', '<i2'), ('q', '<i2')])

np.frombuffer complains if there aren't enough bytes, but fromfile does not.

Write a small array:

In [229]: arr = np.zeros(3, dtype=datatype)
In [230]: arr
Out[230]: array([(0, 0), (0, 0), (0, 0)], dtype=[('i', '<i2'), ('q', '<i2')])
In [231]: arr.tofile('test.txt')

Read it (again no complaints if there aren't S elements):

In [234]: np.fromfile('test.txt', dtype=datatype, count=S)
Out[234]: array([(0, 0), (0, 0), (0, 0)], dtype=[('i', '<i2'), ('q', '<i2')])

Read with context:

In [235]: with open('test.txt','rb') as f:
     ...:     print(np.fromfile(f,dtype=datatype))
     ...: 
[(0, 0) (0, 0) (0, 0)]

Read, but after reading to the end, the result is empty:

In [236]: with open('test.txt','rb') as f:
     ...:     f.read()
     ...:     print(np.fromfile(f,dtype=datatype))
     ...: 
[]

I suspect this last is what's happening in your code.

Or the use a py2.7 might be causing problems. It's been a long time since I used it, and I don't recall what migration problems there were, especially regarding file reading.