Search code examples
c#.netmultithreading.net-coretask-parallel-library

Parallel Tasks - Run Parallel threads but dont wait for other Tasks to complete and get the latest data from database


I am using Task library for running multiple tasks parallelly, and I am using below code to wait for other tasks to complete.

Parallel.ForEach(DatafromDB,
  item => { DownloadSSR(mediator, sourcePath, item,stoppingToken).Wait();
});

Using this I felt like I am not really doing parallelly, because say for Eg: 3 tasks, Task1, Task2 and Task3.

Task1 say takes 15 mins, and other tasks take 1 min respectively, in this case we are waiting for all the Tasks to complete.

To Achieve this I got one more option to use - "MaxDegreeOfParallelism"

Parallel.ForEach(DatafromDB,
  new ParallelOptions { MaxDegreeOfParallelism = 3}, 
  item => { DownloadSSR(mediator, sourcePath, item,stoppingToken).Wait();
});

Now tasks are running parallelly but its taking the data from DB only for the first time.

Eg: if there are 3 data i got from DB then same data source is being used for running tasks parallely.

Here what i want to achieve is -

  1. I want to run the tasks parallelly.
  2. I don't need to wait for other tasks to complete
  3. If one task completed, assign other work to that task (go to Database and get data and assign new work)

Solution

  • Thanks for all the responses.

    Actually I wanted fire and forget mechanism for my requirement, and i know it will put some load on my server, but its fine for me as i am limiting number of threads to be 3 (using maximum degree of parallelism) and i thought Hangfire is one of the best for this requirement.