Exchanging big lists between C# and F#

Question

I have big data files of historical stock data, that I want to load over a C# GUI and then process with different algorithms in F#. Additional values are appended to lists over time.

The lists are two-dimensional and in the form

[[id, Open,High,Low,Close], [id, Open,High,Low,Close], ...]

The F# code is in a library and cannot work with C# lists (I cannot get the values of the second dimension of the list in F#). Converting the entire list every time a F# operation takes place, is too expensive. The values cannot be stored in a F# list either, because C# can't handle them.

Proposed solutions are:

1. either storing an F# list in C#, just for storage purposes since the library can't store values, together with a C# list containing the same values, converting parts of the list as needed. The main problem here is that the files can be multiple GB in size.

2. reading the values that are needed directly from a file in F# and saving calculated values in additional files. The problem here is the efficient organisation of the files, so that a selection of lines (e.g. for a moving average calculation) to load can be done quickly.

Answer 1

Does the inner sequence need to be a list? If it always has the same elements (OHLC) it would be better to use a tuple (or a custom type) since it makes explicit its fixed nature. I would probably do something like this:

module Stock =
  let (!) s = DateTime.Parse(s)

  let getOHLC() : unit -> seq<DateTime * float * float * float * float> =
    seq [
      !"18-Dec-12",27.25,27.63,27.14,27.56
      !"17-Dec-12",26.79,27.22,26.68,27.10
      !"14-Dec-12",27.11,27.13,26.70,26.81
      //...
    ]

If you really need each element to be a list, you can pipe it through Seq.map seq.

The seq function is defined as:

let seq (x:seq<'T>) = x

so it only upcasts, not copy.