Importing large textfile to matlab (including comments; textscan)

Question

Im trying to import big txt files (>1gb) to matlab.

this is the data structure:

667.55535   -0.00   0.000   0.0158
667.5554    -0.01   -0.000  0.0158
667.55545   -0.01   4.037   10.0000
667.5555    -0.00   4.000   10.0000 #1 Trigger Camera 10 Hz #2 Trigger Camera 10 Hz 
667.55555   -0.00   4.000   10.0000
667.5556    -0.01   4.000   10.0000

I use the textscan function:

segarray = textscan(file_id, '%f %f %f %f', blocksize, 'delimiter','\n', 'commentStyle', '#');

works pretty well, but i NEED the comments marked with ' #' if i change the format string to '%f %f %f %f %s' and remove the 'commentStyle', '#' option every second line is read as a single string :/

any ideas?

Answer 1

You can use

segarray = textscan(fid, '%f %f %f %f %[^\n]');

to achieve what you want (so without any options to textscan()). The last format character means that textscan will match any trailing character(s) that are not newline.

This results in:

test.txt:

667.55535   -0.00   0.000   0.0158
667.5554    -0.01   -0.000  0.0158
667.55545   -0.01   4.037   10.0000
667.5555    -0.00   4.000   10.0000 #1 Trigger Camera 10 Hz #2 Trigger Camera 10 Hz 
667.55555   -0.00   4.000   10.0000
667.5556    -0.01   4.000   10.0000
667.5555    -0.11   4.000   12.0000 #1 Trigger Camera 11 Hz #2 Trigger Camera 12 Hz 
667.5557   -0.00   4.000   10.0000
667.556    -0.01   4.000   10.0000
667.55855   -0.00   4.000   10.0000
667.5596    -0.01   4.000   10.0000
667.55105   -0.00   4.000   10.0000
667.5511    -0.01   4.000   10.0000

segarray{:}:

[first three columns omitted for brevity] 

ans =
    0.0158
    0.0158
   10.0000       
   ...       % fourth column abbreviated
   10.0000
   10.0000

ans = 
    ''
    ''
    ''
    '#1 Trigger Camera 10 Hz #2 Trigger Camera 10 Hz '
    ''
    ''
    '#1 Trigger Camera 11 Hz #2 Trigger Camera 12 Hz '
    ''
    ''
    ''
    ''
    ''