How does Flinks Collector.collect() handle data?

Question

Im trying to understand what Flinks Collector.collect() does and how it handles incoming/outgoing data:

Example taken from Flink DataSet API:

The following code transforms a DataSet of text lines into a DataSet of words:

DataSet<String> output = input.flatMap(new Tokenizer());

public class Tokenizer implements FlatMapFunction<String, String> {
  @Override
  public void flatMap(String value, Collector<String> out) {
    for (String token : value.split("\\W")) {
      out.collect(token);
    }
  }
}

So the text Lines get split into tokens and each of them gets "collected". As intuitive as it might sound but im missing the actual dynamics behind Collector.collect(). Where is the collected data stored before it gets assigned to output i.e does Flink put them in some sort of Buffer? And if yes, how is the data transferred to the network?

Answer 1

from the official source code documentation.

Collects a record and forwards it. The collector is the "push" counterpart of the {@link java.util.Iterator}, which "pulls" data in.

So, it receives a value and stores one or more values into the Iterator. Then pushes to the next operator. But this is a matter of the network stack/ buffers.