Search code examples
rdisk.frame

How should we choose the compression rate with rbindlist.disk.frame?


It's set to 50 by default on a scale of 1 to 100.

I have an especially large disk frame and I'm considering using a high number.

What are the important trade-offs to consider?


Solution

  • The compression uses the fst compressions. You are free to experiment but {disk.frame} might change the backend in future to other format other than fst.

    100 compression will result in smaller files but longer time to decompress. Your mileage definitely varies. There is no hard and fast rule as the compression that will result depends on the type of data and the content of the data.