storing parsed csv using CsvHelper as suitable datatype

Question

How do I go about storing the parsed csv in a suitable datatype to perform calculations, view it etc. I find a method called records.ToList that seems promisingbut I can't get the syntax right.

    class XLOperations
{
    public static void ParseCSV() {
        using (var reader = new StreamReader("data.csv"))
        using (var csv = new CsvReader(reader, System.Globalization.CultureInfo.InvariantCulture))
        {
            var records = csv.GetRecords<dynamic>();
            //here I want to store the parsed csv as a List<> or other suitable datatype to work with
        }
    }
}

}

Answer 1

The best place to start would be with the CsvHelper Getting Started page.

What can be confusing is that csv.GetRecords<dynamic>(); returns IEnumerable<dynamic> which will yield records as you iterate over the IEnumerable object. What this means is you are not getting a list of items yet, what you are getting is an enumerator which keeps track of where you are in getting records and only returns them as you ask for them.

Often you are getting csv files from disk and this can be a costly operation. Say you just wanted to get the first 5 records from a file that had 50 million records. You could call

var records = csv.GetRecords<dynamic>().Take(5).ToList();

This would iterate through the first 5 records on disk and then return a List<dynamic> of 5 records. If you flipped ToList() and Take(5), the means of getting the data would be very different.

var records = csv.GetRecords<dymanic>().ToList().Take(5);

This would iterate through all 50 million records on disk and bring them into memory. Then it would take the first 5 of those records and return them to you as IEnumerable<dynamic>

The difference could be a fraction of a second vs minutes in getting those 5 records and considerably more memory used.