Limiting number of threads when dequeuing from a BlockingCollection

Question

I am currently using this to limit the number of processing threads:

Dim ListOfFiles New List(of String) From {...}
Dim parallelOptions As ParallelOptions = New ParallelOptions With {.MaxDegreeOfParallelism = 6}
Parallel.ForEach(ListOfFiles, parallelOptions, Sub(F)
                                                    'Process one file
                                               End Sub)

which works well because I can control the number of concurrent threads to suit other processing going on simultaneously.

But I need to keep adding items to the list. Instead of a fixed size list, I want to use a ConcurrentQueue so that I can continue enqueuing a list of files (from a Producer thread) while the parallel Consumer processing is going on. Clearly Parallel.ForEach wont work. I could use a BlockingCollection but I cant determine how best to only dequeue items as threads become available and block when they are all busy. Sitting in a loop and counting threads seems ugly.

Any thoughts would be welcome.

Answer 1

Thanks to all who commented.

I ended up using GetComsumingPartitioner based on the blog article by Stephen Toub here: https://devblogs.microsoft.com/pfxteam/parallelextensionsextras-tour-4-blockingcollectionextensions/

This worked out great.