I am trying to save roughly 20,000 records and it takes an unforgivable amount of time to finish. What is the more optimal way of achieving this?
Here's what I currently have:
public async Task SaveOrUpdateItemAsync(List<DepartmentalizedItem> departmentalizedItems)
{
using(WarehouseContext dbContext = new WarehouseContext())
{
using(SemaphoreSlim throttler = new SemaphoreSlim(20))
{
var tasks = departmentalizedItems.Select(async item =>
{
await throttler.WaitAsync();
if (item.PK_DepartmentalizedItemId == 0)
dbContext.DepartmentalizedItems.Add(item);
else
{
var deptItem = await dbContext.DepartmentalizedItems.FindAsync(item.PK_DepartmentalizedItemId);
dbContext.Entry(deptItem).CurrentValues.SetValues(item);
}
throttler.Release();
});
await Task.WhenAll(tasks);
}
await dbContext.SaveChangesAsync();
}
}
I've also tried Parallel.ForEach but I'm having synchronization issues.
Thank you.
Thank you for all answers, comments and suggestion. All of those has been taken into consideration and I have finally arrived to a more optimal way of saving/editing 20,000 records in bulk. As suggested by Hasse, I took a peek on BulkInsert, but this is a paid library, thus I tried to find any alternatives and fortunately I found N.EntityFramework.Extensions.
Here's what solved my issue:
public async Task BulkSaveAndUpdateAsync(List<DepartmentalizedItem> departmentalizedItems, CancellationToken cancellationToken)
{
using (WarehouseContext dbContext = new WarehouseContext())
{
var toAddItems = departmentalizedItems.Where(i => i.PK_DepartmentalizedItemId == 0);
var toUpdateItems = departmentalizedItems.Where(i => i.PK_DepartmentalizedItemId > 0);
await dbContext.BulkInsertAsync(toAddItems, cancellationToken);
await dbContext.BulkUpdateAsync(toUpdateItems, cancellationToken);
}
}