I am building a consumer and I want to ensure that every message, from a Kafka topic, is read and processed accordingly. While the auto commit feature is appealing, I would want to have full control of the offsets -- meaning, I want to only commit the offsets if the processing went well.
However, I'm not sure about the downsides of turning auto commit to false. What are the PROs and CONs of turning auto commit off? Is it gonna perform slower? Will multiple consumers, with the same consumer name have a problem with this implementation?
Performance really depends on how you implement it - you can commit offsets synchronously, or asynchronously, having different guarantees about how sure you want to be about the fact that offsets are committed. Usually when you don't commit offsets manually, you have more control about it - like you want. But there are some caveats, like, commitSync
or commitAsync
are committing offsets that are obtained by poll, so if you commit in the middle of the processing loop, and application crashes, then you'll lose data.
"Kafka. The Definitive Guide" has big section about commits & offsets - I recommend to read it - Confluent distributes the PDF of the book for free.