c# - splitting a large list into smaller sublists

Question

Fairly new to C# - Sitting here practicing. I have a file with 10 million passwords listed in a single file that I downloaded to practice with.

I want to break the file down to lists of 99. Stop at 99 then do something. Then start where it left off and repeat the do something with the next 99 until it reaches the last item in the file.

I can do the count part well, it is the stop at 99 and continue where I left off is where I am having trouble. Anything I find online is not close to what I am trying to do and anything I add to this code on my own does not work.

I am more than happy to share more information if I am not clear. Just ask and will respond however, I might not be able to respond until tomorrow depending on what time it is.

Here is the code I have started:

using System;
using System.IO;


namespace lists01
{
class Program
{
    static void Main(string[] args)
    {
        int count = 0;
        var f1 = @"c:\tmp\10-million-password-list-top-1000000.txt";           
        {                
            var content = File.ReadAllLines(f1);
            foreach (var v2 in content)
            {
                count++;
                Console.WriteLine(v2 + "\t" + count);
            }
        }
    }
}
}

My end goal is to do this with any list of items from files I have. I am only using this password list because it was sizable and thought it would be good for this exercise.

Thank you Keith

Answer 1

Here is a couple of different ways to approach this. Normally, I would suggest the ReadAllLines function that you have in your code. The trade off is that you are loading the entire file into memory at once, then you operate on it.

Using read all lines in concert with Linq's Skip() and Take() methods, you can chop the lines up into groups like this:

var lines = File.ReadAllLines(fileName);
int linesAtATime = 99;

for (int i = 0; i < lines.Length; i = i + linesAtATime)
{
    List<string> currentLinesGroup = lines.Skip(i).Take(linesAtATime).ToList();
    DoSomethingWithLines(currentLinesGroup);
}

But, if you are working with a really large file, it might not be practical to load the entire file into memory. Plus, you might not want to leave the file open while you are working on the lines. This option gives you more control over how you move through the file. It just loads the part it needs into memory, and closes the file while you are working on the current set of lines.

List<string> lines = new List<string>();
int maxLines = 99;
long seekPosition = 0;
bool fileLoaded = false;
string line;

while (!fileLoaded)
{
    using (Stream stream = File.Open(fileName, FileMode.Open))
    {
        //Jump back to the previous position
        stream.Seek(seekPosition, SeekOrigin.Begin);

        using (StreamReader reader = new StreamReader(stream))
        {
            while (!reader.EndOfStream && lines.Count < maxLines)
            {
                line = reader.ReadLine();
                seekPosition += (line.Length + 2); //Tracks how much data has been read.
                lines.Add(line);
            }
            fileLoaded = reader.EndOfStream;
        }
    }

    DoSomethingWithLines(lines);
    lines.Clear();
}

In this case, I used Stream because it has the ability to seek to a specific position in the file. But then I used StreaReader because it has the ReadLine() methods.