TPL dataflow that receives a collection and calls its linked block for each element

Question

sorry if there is already a similar question, I can't find it.

I have the following situation:

1. I have to do some processing on images and the TPL Dataflow fits in nicely here, because it allows me to easily do different parts of my workflow in parallel and separate code in logical units
2. There is a function I don't have control over that returns a list of images. It is used as a second node in my mesh (or pipeline, to be more precise)
3. All my other nodes in Dataflow mesh work with a single image, so the node that comes after the I one I mentioned in the second bullet expects to get one image (this is important for parallelism reasons)

Is there a block (or some other solution) I can use that would take an input of type IEnumerable<T> or something similar and forward each element of that IEnumerable to a block that expects to receive T?

I don't want to reinvent the wheel, so I wanted to check is there an easy solution before digging into the API and trying to write a custom block. Also, it is important that error and completion are propagated to the end of the pipeline.

Thanks for the answer!

Answer 1

No need for a custom block. Your're looking for the TransformManyBlock. Here's a simple demo:

public async Task TransformManyExample() {
    var data = Enumerable.Range(0, 10).ToList();
    var block1 = new TransformManyBlock<IEnumerable<int>, int>(x => x);
    var block2 = new ActionBlock<int>(x => Console.WriteLine(x.ToString()));
    block1.LinkTo(block2, new DataflowLinkOptions() { PropagateCompletion = true });
    block1.Post(data);
    block1.Complete();
    await block2.Completion;
}