Search code examples
apache-flinkflink-streaming

Flink broadcast state - RocksDB state backend


In the documentation for the broadcast pattern, it is mentioned that there is not RocksDB state backend:

No RocksDB state backend: Broadcast state is kept in-memory at runtime and memory provisioning should be done accordingly. This holds for all operator states.

How does this affect savepoint behavior if the application uses rocksdb as the state backend? Does this mean that state is not stored during savepoint and hence not restored?


Solution

  • Broadcast state is included in savepoints and checkpoints.

    Flink distinguishes between keyed and non-keyed state. All of the state managed by Flink, both keyed and non-keyed, is included in savepoints and checkpoints.

    Broadcast state is a kind of non-keyed state, and like all non-keyed state, is not stored in RocksDB. When RocksDB is used as the state backend, this means that the working state for keyed state is kept in RocksDB, rather than on the heap. When a state snapshot is taken (i.e., a savepoint or checkpoint), this state from RocksDB plus all of the non-keyed state is written into the snapshot.