How can I limit concurrent HTTP requests and handle timeouts in .NET Core?

Question

I'm working on a .NET Core app where I need to make a bunch of HTTP requests in parallel. The tricky part is that the number of requests can vary a lot, depending on user input—sometimes it's as few as 5, other times it could be up to 50.

Right now, I'm using Task.WhenAll to handle them like this:

var tasks = urls.Select(url => httpClient.GetStringAsync(url)).ToArray();
var results = await Task.WhenAll(tasks);

This works fine for small numbers, but I'm worried about the app getting overwhelmed when there are too many requests at once. I also want to make sure that if one request times out or fails, the others can still finish properly.

A couple of things I’m wondering:

Is there a way to limit how many requests run at the same time, like maybe 10 at a time?

If one of the requests takes too long, how do I make sure it times out without affecting the others?

Is retrying failed requests a good idea, or is there a better approach to handle that?

I’m not sure if Task.WhenAll is the best way to handle this or if there's a better pattern I should be looking at. Any suggestions would be awesome!

Answer 1

There are probably a bunch of ways you can do this (PLINQ, Polly with BulkHead pattern, ...). A bare bone approach matching your example code would just use SemaphoreSlim

var tasks = urls.Select(url => httpClient.GetStringAsync(url)).ToArray();
var results = await Task.WhenAll(tasks);

var semaphore = new SemaphoreSlim(5);
var tasks = urls.Select(async url =>
{
    await semaphore.WaitAsync();
    try
    {
        return await httpClient.GetStringAsync(url);
    }
    finally
    {
        semaphore.Release();
    }
}).ToArray();

var results = await Task.WhenAll(tasks);

Depending on your needs you probably should also look into handling timeouts (Hint: CancellationTokenSource) and retrying failed requests (Polly library)