Search code examples
c#asp.net-coreasynchronous.net-8.0

Asynchronously generate and add POCO to List<T>


I'm executing a query that returns approximately 11,000 rows. Each one of these rows is mapped to a unique POCO that I wrote a lightweight ORM to handle creating, called ToCustomObject<T>. Right now, I have an already async method that gets the results of this query and puts them into a DataTable. I then loop through all of the rows and call ToCustomObject on them to generate my object, and then add it to a List<T>. The query itself only takes half a second, but the loop to generate all my POCOs takes multiple seconds.

public async Task<List<Foo>> GetFoosAsync()
{
    //empty list to populate and return
    List<Foo> foos = [];

    //example, this is not how the query actually executes, do not need assistance with this part
    string query = "select t.Field1, t.Field2 from Table";
    DataTable results = await customDbHandlerThatsNotRelevant.ExecuteQueryAsync(query);

    //async way to handle this?
    foreach (DataRow row in results.Rows)
        foos.Add(row.ToCustomObject<Foo>());

    return foos ;
}

I had thought about maybe using Task.WhenAll() to put all the generations into a task list and execute that way, but foos.Add() is not awaitable. I also looked into Parallel.ForEach(), but I'm not as familiar with that and from what I could tell, would not accomplish what I'm trying to do.

TL;DR: How can I make all occurrences of foos.Add(row.ToCustomObject<Foo>()); occur asynchronously?


Solution

  • Well it greatly depends. Are you seeing performance bottleneck in that loop? If yes, then maybe it is worth parallelizing.

    But generally, it is not meant to be executed asynchronously, as your method ToCustomObject is not async, so you would need to wrap each execution in Task, which is already overhead. Doing that for all item in a list may even downgrade your performance!

    BUT, if you are seeing performance issues, I can suggest couple of ways:

    1. wrapping ToCustomObject call in Task, creating Task for each item and awaiting them all at one
    2. use Parallel.ForEach
    3. if ToCustomObject is performing badly, maybe there's some synchornous code inside of it that can be done asynchoronously? Then you could make async version of ToCustomObject

    Of course, you need to take care of your List beong accessed potentially from multiple threads, do you need to use some synchronization or threads safe collection.