How to Use all processor cores for x64 application

Question

I have got a simple code for compare two arrays:

static void Main(string[] args)
    {
        String directory = @"C:\path\";
        String[] linesA = File.ReadAllLines(Path.Combine(directory, "array1.txt"));
        String[] linesB = File.ReadAllLines(Path.Combine(directory, "array2.txt"));

        var onlyB = linesA.Where(a => linesB.Any(b => a.ToLower().Contains(b.ToLower())));

        File.WriteAllLines(Path.Combine(directory, "Result.txt"), onlyB);
    }

Is there any method to use all cores of processor to compare those two arrays?

Answer 1

If you actually want to speed this up (rather than just use more memory) there are a few things you can to.

1. Instead of using File.ReadAllLines() for both files (which reads the entire file into memory), use File.ReadLines() for the file used in the outer loop. This will only read that file line-by-line and avoid a potentially large amount of memory use.
2. Instead of using ToLower() on the strings multiple times, use string.IndexOf() like so: var onlyB = linesA.Where(a => linesB.Any(b => a.IndexOf(b, StringComparison.OrdinalIgnoreCase) >= 0));
3. Use .AsParallel() to parallelise part of the loop.

Putting this all together:

var linesA = File.ReadLines(Path.Combine(directory, "array1.txt"));
var linesB = File.ReadAllLines(Path.Combine(directory, "array2.txt"));

Stopwatch sw = Stopwatch.StartNew();

var onlyB = linesA.AsParallel().Where(a => linesB.Any(b => a.IndexOf(b, StringComparison.OrdinalIgnoreCase) >= 0));

After making these changes, my test program went from taking ~3.8s to ~0.6s, and used a lot less memory.