Search code examples
apache-flinkflink-streaming

will flink copy element in each stream or use the same object?

If two operator are processing the same upstream, does each operator receives a copy of each element or they are processing the same element. For example, in the follow code:

DataStream<Data> stream=...; //init a stream of data
stream.process(ProcessFunctionA); //stream was processed by function A
stream.process(ProcessFunctionB); //stream was processed by function A

If ProcessFunctionA change the Data, will ProcessFunctionB see the changes?

Solution

  • By default, Flink makes defensive copies in situations like this. But if you understand the possible pitfalls of doing so, you can use

    ExecutionConfig#enableObjectReuse()

    to enable object reuse, which can be a useful optimization, since it avoids unnecessary copies and their eventual garbage collection.

    See Flink, rule of using 'object reuse mode' for some guidelines on how to use this safely.