The wait operation timed out exception with Linq

Question

I need to develop simple windows desktop app that will load CSV file in single table MDF, generate some reports from it and then delete data. CSV file could contain millions of records...

When i try to execute code to show data in dbGridView, exception "The wait operation timed out" is thrown.

  db dbEntities = new db();

    var ds = (from tbl in db.tbl_csv
              orderby f1
              select new
              {
                  f1= tbl.f1,
                  f2 = tbl.f2,
                  f3= tbl.f3,
                  f4= tbl.f4,
                  f5= tbl.f5,
                  f6= tbl.f6,
                  f7= tbl.f7,
                  f8= tbl.f8
              }
                  );
    var bds = ds.ToList();
    return Helpers.ToDataSet(bds);

This works if csv contains small amount of data, but when it has more than 70-80k records, exception is thrown...

Is there any workaround for this?

Answer 1

Read n number of records at a time. e.g.

int totalRecords = ds.Count();
int n = 100;
int chunksRead = 0;
int recordsRead = 0;

while(recordsRead < totalRecords)
{
    ds.Skip(recordsRead).Take(n);
    // process n records
    ...
    chunksRead++;
    recordsRead = chunksRead * n;
}