I'm new to flink and I have some confusion about the state backend configuration.
As far as I know, RocksDB saves all of the application's state on the filesystem. I use s3 to store the state, so I configured both state.checkpoints.dir and state.savepoints.dir pointed to my s3 bucket. Now I see that there is another option related to RocksDB storage called state.backend.rocksdb.localdir. What is the purpose of this?(I saw I can't use s3 for this) Also, if RocksDB uses the local machine storage for something, what will be when I use Kubernetes and my pod suddenly failed? should I use persistent storage?
Another thing, I'm not sure I understood all the state things correctly. Does the checkpoint save all of my state? For example, when I use AggregationFunction and the application failed, when the application restored, does the aggregated value for each key is restored?
Each of Flink's state backends keeps its working state somewhere local to each worker, while persisting the checkpoints somewhere durable, such as S3. With the heap-based state backend, the working state is stored as objects on the JVM heap, while with RocksDB the working state is stored as serialized bytes on the local disk (with an in-memory, off-heap cache). For performance reasons you don't want to use S3 (or even network-attached storage) for state.backend.rocksdb.localdir. Use local SSD storage if you can.
Flink doesn't rely on the local rocksdb storage surviving failures, just as it doesn't expect state on the heap to survive a failure, so you can safely use ephemeral storage as the rocksdb.localdir. When the state does need to be recovered, the latest checkpoint is sufficient. (But the copy on the local disk can be used as an optimization, avoiding the need to read from the DFS: see the docs on state.backend.local-recovery for details.
During recovery the aggregated value for each key in an AggregationFunction will be restored, should your application fail. The checkpoints include everything, including state kept by the sources and sinks, windows, timers, ProcessFunctions, RichFunctions, etc.