How to prevent to create more threads than the available number of hardware threads?

Question

I have the following method below that used to initialize and start log files processing by creating a different task for each file. In the Task Manager I can see this function is creating about one hundred threads this way.

public async Task LogProcessing(DescriptorList files, CancellationToken ct)
{
    var tasks = new List<Task>();
    foreach (var file in files)
        tasks.Add(new Task(() =>
        {
            ParsingFile(file, ct);
        }));

    foreach (var task in tasks) { task.Start(); }
   
    await Task.WhenAll(tasks);       
}

How can I prevent creating more threads than the available number of hardware threads? Thanks.

Answer 1

If you are using .NET 6.0 or later you could use Parallel.ForEachAsync() to do this:

await Parallel.ForEachAsync(
    files,
    new ParallelOptions { MaxDegreeOfParallelism = Environment.ProcessorCount, CancellationToken = ct },
    async (file, cancellation) =>
    {
        await Task.Run(() => ParsingFile(file, cancellation), cancellation);
    });

Note the use of MaxDegreeOfParallelism = Environment.ProcessorCount to limit the number of concurrent threads to the (logical) processor count. This is actually usually the default, so you may not need to set this at all, but some folks might like to do so for clarity.

According to the documentation "Generally, you do not need to modify this setting" but you should look at this answer from Theodor Zoulias before using the default. (I generally don't use the default myself).

I recommend reading the documentation on MaxDegreeOfParallelism and deciding for yourself whether to specify it or not.

As per TheodorZoulias's comments below, you can write this more simply as:

await Parallel.ForEachAsync(
    files,
    new ParallelOptions { MaxDegreeOfParallelism = Environment.ProcessorCount, CancellationToken = cancelSource.Token },
    (file, cancellation) =>
    {
        ParsingFile(file, cancellation);
        return ValueTask.CompletedTask;
    });

However, if ParsingFile() was itself async you would of course use the first version.