Search code examples
matlabfile-iobinaryfilesbigdatamemory-mapped-files

how to read interleaved data with memmapfile()?


I have a large (4GB+) data file I'd like to access. It contains samples i of a number of different signals {a, b, c} as follows:

a_1 b_1 c_1 a_2 b_2 c_2 .... a_n b_n c_n

I would like to use memmapfile to retrieve, say, the a stream. Since I know the number of signals and the number of samples, I tried:

m = memmapfile('data.dat','Format',{'int16',[nSignals 1],'sid'},'repeat',nSamples);

but this returns the useless m field

Data: nSamples x 1 struct array with fields:
          sid

Of course the following works fine, but is very slow:

m = memmapfile('data.dat','Format','int16');
a = m.Data(1:nSignals:end);

How can I recover a without having to access the full data matrix?


Solution

  • How about using FREAD and specifying the appropriate skip value. The following will read the signal a in one go:

    % 3 interleaved signals each of type int16
    nSignals = 3;
    
    % amount of bytes to skip after reading each sample
    szINT16 = 2;                   % sizeof(int16)=2
    skipBytes = (nSignals-1)*szINT16;
    
    % number of samples in each signal (Inf to read all samples)
    nSamples = Inf;
    
    fid = fopen('data.dat','rb');
    a = fread(fid, nSamples, '*int16', skipBytes);
    fclose(fid);
    

    You could do the same for the other two signals, you just have to seek the right start location:

    fseek(fid, szINT16*1, 'bof');
    b = fread(fid, nSamples, '*int16', skipBytes);
    
    fseek(fid, szINT16*2, 'bof');
    c = fread(fid, nSamples, '*int16', skipBytes);