Search code examples
kdb+tickerchunksq-lang

Q/kdb+ read log files by chunk

I have a task that requires me to load log files that is too big for my memory to let it load at once. Therefore I need way to load the file by different chunks..

I know that:

-11!(n; filename)

loads first n chunk from a file, but how do I load the rest of the chunks??

Thanks!

Solution

  • Your log file probably calls the function upd if it is a typical tickerplant setup. Override that function to ignore the initial values on the second run, without inserting them into the table allowing you to split the loading into parts.

    upd:{i+:1; if[i within 100000 200000; ignore x; storeSomewhere x]};

    Then play the whole file -11!. Control what portions are actually stored by changing the condional in the if clause.