Search code examples
c#asp.net-coreparallel-processingtask-parallel-library

What is the best way to call API calls in parallel in .NET Core, C#?


I would like to call my API in parallel x number of times so processing can be done quickly.

I have three methods below that I have to call APIs in parallel. I am trying to understand which is the best way to perform this action.

Base Code:

var client = new System.Net.Http.HttpClient();
client.DefaultRequestHeaders.Add("Accept", "application/json");

client.BaseAddress = new Uri("https://jsonplaceholder.typicode.com");
var list = new List<int>();

var listResults = new List<string>();
for (int i = 1; i < 5; i++)
{
    list.Add(i);
}

First method using Parallel.ForEach:

Parallel.ForEach(list,new ParallelOptions() { MaxDegreeOfParallelism = 3 }, index =>
{
    var response = client.GetAsync("posts/" + index).Result;

    var contents =  response.Content.ReadAsStringAsync().Result;
    listResults.Add(contents);
    Console.WriteLine(contents);
});

Console.WriteLine("After all parallel tasks are done with Parallel for each");

Second method with Tasks. I am not sure if this runs parallel. Let me know if it does:

var loadPosts = new List<Task<string>>();
foreach(var post in list)
{
    var response = await client.GetAsync("posts/" + post);

    var contents = response.Content.ReadAsStringAsync();
    loadPosts.Add(contents);
    Console.WriteLine(contents.Result);
}

await Task.WhenAll(loadPosts);

Console.WriteLine("After all parallel tasks are done with Task When All");

Third method using Action Block - This is what I believe I should always do but I want to hear from community:

var responses = new List<string>();

var block = new ActionBlock<int>(
    async x => {
        var response = await client.GetAsync("posts/" + x);
        var contents = await response.Content.ReadAsStringAsync();
        Console.WriteLine(contents);
        responses.Add(contents);                
    },
    new ExecutionDataflowBlockOptions
    {
        MaxDegreeOfParallelism = 6, // Parallelize on all cores
    });

for (int i = 1; i < 5; i++)
{
    block.Post(i);
}
           
block.Complete();
await block.Completion;

Console.WriteLine("After all parallel tasks are done with Action block");

Solution

  • Approach number 2 is close. Here's a rule of thumb: I/O bound operations=> use Tasks/WhenAll (asynchrony), compute bound operations => use Parallelism. Http Requests are network I/O.

                foreach (var post in list)
                {
                    async Task<string> func()
                    {
                        var response = await client.GetAsync("posts/" + post);
                        return await response.Content.ReadAsStringAsync();
                    }
    
                    tasks.Add(func());
                }
    
                await Task.WhenAll(tasks);
    
                var postResponses = new List<string>();
    
                foreach (var t in tasks) {
                    var postResponse = await t; //t.Result would be okay too.
                    postResponses.Add(postResponse);
                    Console.WriteLine(postResponse);
                }