Search code examples
.netcsvf#deedle

Deedle throws System.OutOfMemoryException when reading large csv file

I was very excited to find Deedle few days ago which seems to be great tool for data analysis in NET world. However some of my data are in csv files and Frame.ReadCsv method throws System.OutOfMemoryException for files about 200MB or larger. I played with Frame.ReadCsv parameters with no success. I can pre-process data in some cases before loading into dataframe, but it would be nice to be able handle them directly in Deedle. R has no issues loading these files into dataframe. Any recommendation?

I come across BigDeedle example, but it does not look trivial and I wonder if there are simpler solution which does not require additional code. Thanks!

Solution

  • Check your FSI settings for 64 bit with

    let is64Bit = (System.IntPtr.Size = 8)

    And configure Visual-Studio to switch FSI to 64 bit, here:

    Tools -> Options -> F# Tools -> F# Interactive