Search code examples
apache-kafkakafka-consumer-apikafka-producer-api

Force consumer to read only those messages that came in topic after consumer came to life


Is there any way that we can force the consumer to read the latest? e.g. Let's say I have one Topic t1 and consumer C1 in the consumer group CG1. If t1 has 10 messages (m0 to m10) in it, as soon as C1 comes to life it reads first 5 messages and then crashes, now before crashing let's say it committed offest to 5. In the meanwhile, the topic received 5 more messages (m11 to m15). Now C1 comes back up and after that t1 receives 5 more messages (m16 to m20). I want C1 to force start reading from m16 onwards i.e. from the message that t1 received after C1 came back up. Because of the committed offset, it will start reading from m6 onwards. I am OK with C1 to lose message when it was down. Is there any way I can configure my consumer for this use-case.


Solution

  • Setting enable.auto.commit = false and auto.offset.reset = latest on my consumer works for this usecase.