Search code examples
amazon-s3apache-flinkflink-streaming

Trying to enable Flink Entropy with S3, but entropy key is not generated

I am trying to enable the entropy feature provided by Flink, but the at the end the _entropy_ is not resolved (i mean there is no "key" generated)

I have the following configuration:

s3.access-key   my-key
s3.secret-key   ******

s3.entropy.key  _entropy_
s3.entropy.length   8

state.backend   filesystem
state.backend.incremental   true
state.backend.local-recovery    true

state.checkpoints.dir   s3p://<my-bucket>/_entropy_/flink/checkpoints/stage
state.checkpoints.num-retained  2
state.savepoints.dir    s3p://<my-bucket>/flink/savepoints/stage

when I open my-bucket I see the folder _entropy_ instead of a random 8 characters folder.

For the declaration in the code :

env.enableCheckpointing(10000L, CheckpointingMode.EXACTLY_ONCE)
   .getCheckpointConfig()     
   .enableExternalizedCheckpoints(ExternalizedCheckpointCleanup.DELETE_ON_CANCELLATION);
env.getCheckpointConfig().setMinPauseBetweenCheckpoints(3000L);

PS: I'm using Flink 1.11.2

Solution

  • There was a regression in Flink 1.11.2 that broke this feature; this was fixed in 1.11.4. See FLINK-22081 for details.