How to disable PLINQ partitioning/batching

Question

PLINQ internally batches the items that it takes in order to reduce overhead. Since my items are very memory intensive I'd like to minimize any buffering that exists in the PLINQ query pipeline. How can I disable partitioning/batching entirely?

The code looks like this:

var myItems = Enumerable.Range(0, 10000000).Select(_ => new byte[1 << 30]);
var results =
 myItems
 .AsParallel()
 .WithMaxDegreeOfParallelism(4)
 .Select(F)
 .ToList();

In this code I'd expect the maximum number of (big) items that are ineligible for garbage collection to be 4.

Answer 1

What you want to do is to create a Paritioner over your source, which specifies NoBuffering, and then use that in your PLINQ query:

var myItems = Enumerable.Range(0, 1000).Select(_ => new byte[1 << 30]);
var myPartitioner = Partitioner.Create(myItems, EnumerablePartitionerOptions.NoBuffering);
var results =
    myPartitioner
    .AsParallel()
    .WithDegreeOfParallelism(4)
    .Select(F)
    .ToList();

This query works for me, but throws OutOfMemoryException if I skip the partitioner (just like your original query).