Search code examples
apache-kafkaapache-flink

Not committing a message from Apache Kafka if exception occurs while operations in Apache Flink


Sometimes, while processing and during sink I am getting few exceptions because of network. I want these message not to be committed so that they can be re processed sometimes in future.

Is there any way to achieve such functionality where I could ignore that individual message from offset commit, so that it can be processed sometimes in future?

One solution, I am currently following is to sink those messages to other topic which we process later.


Solution

  • Your approach of writing messages with errors during processing to a "dead letter queue" is a common and useful pattern. It's also pretty simple and straightforward. Don't change anything.