Search code examples
filec#-4.0filestreamlarge-filesmemory-mapped-files

How to read very large text file(in gb's) in .net c# and split into small files

I'm tasked with reading a large text files(not XML's) may be in GB's in size. I need to split the file into smaller chunks by checking for the header . Suggest me some method to achieve this. Sample text structure will be

Sample large file
header_start
blaw
blaw
blawasasdasda
header_start
blaw
blawasdasda
blaw

Need to split into

1.txt header_start
blaw
blaw
blawasasdasda


2.txt
header_start
blaw
blawasdasda
blaw

Please help me to achieve this in .net 4.0 with less time

Thanks Vivek

Solution

  • Create a StreamReader for the large file and call ReadLine() in a loop.
    Maintain a StreamWriter for the current output file.
    For each line, check whether it's a header, and, if it is, open a new target file in the StreamWriter. If it isn't, just write that line to the current StreamWriter.