Search code examples
apache-flink

Does all instances (Parellelism) of Async I/O Operator share the same Broadcast State


To know whether broadcast state is shared among all the instances of an Operator.

Operator can be anything, but in my case it is Async I/O Operator.


Solution

  • Each instance of an operator handling broadcast state receives and potentially stores its own copy of that state (and each instance independently checkpoints its broadcast state). If there are multiple sources of broadcast state, the order of processing those broadcast streams in downstream operators is non-deterministic, which could lead to inconsistencies among the various instances.