System.OutofMemoryException while reading a large text file using C#

Question

I have code that reads a text file and populate a .Net datatable. The code works fine when it read a smaller size of text file that has 100,000 lines of data. (see snippet below) When I try to read a larger text file size like 200MB and has 3.6 millions line of data throws me an exception of System.OutofMemoryException. Would like to ask an efficient way of reading a large data into a certain chunks.

        using (var stream = File.Open(filePath, FileMode.Open))
        {
            var content = new StreamContent(stream);
            var fileStream = content.ReadAsStreamAsync().Result;

            if (fileStream == null) throw new ArgumentException(Constants.FileEmptyErrorMessage);

            using (var bs = new BufferedStream(fileStream))
            {
                using (var reader = new StreamReader(bs, Encoding.GetEncoding(Constants.IsoEncoding)))
                {


                    while (!reader.EndOfStream)
                    {
                        var line = reader.ReadLine();
                        if (!String.IsNullOrEmpty(line))
                        {
                            string[] rows = line.Trim().Split(new char[] { ';' }, StringSplitOptions.None);

                            DataRow dr = Table.NewRow();
                            dr[Constants.Percepcion] = rows[0];
                            dr[Constants.StartDate] = DateTime.ParseExact(rows[2].ToString(), "ddMMyyyy",
                                CultureInfo.InvariantCulture);
                            dr[Constants.EndDate] = DateTime.ParseExact(rows[3].ToString(), "ddMMyyyy",
                                CultureInfo.InvariantCulture);
                            dr[Constants.CID] = rows[4];
                            dr[Constants.Rate] = rows[8];

                            Table.Rows.Add(dr);
                        }
                    }
                }
            }
        }

Answer 1

If you alter the default buffer size of your BufferedStream, then it should load the larger files for you with greater efficiency. E.g.

using (var bs = new BufferedStream(fileStream, 1024))
{
    // Code here.
}

You may be able to get away with simply using a FileStream, specifying a buffer size also, rather than a BufferedStream. See this MSDN blog regarding it for further details.