Search code examples
elasticsearchapache-flinkflink-streamingsink

How does Flink sink behave when recovering from a savepoint?


Curious to know how elasticsearch sink connector behaves when a recovery from a savepoint on Flink is completed.

Would it delete/update the documents on ES sink destination by reverting the commits up to the point of savepoint time or should some other mechanism recover ES index for them to sync up again?

I can generalize this question to other sink connectors as well, such as Kafka.


Solution

  • Flink can support exactly once delivery through two-phase commit, and end-to-end exactly once delivery also need connector guarantee

    Kafka Source and Sink support exactly once delivery,but es connector only support at lease once delivery,after es sink is restored from savepoint, there will be no reverting operation