We have a working rabbitmq .implementation , due to volume, we are planning to switch to kafka.
I have a doubt at one point.
In rabbitMQ when the consumer consumes the message from the Q the message goes to a different stage , unacked stage. the client/consumer takes some time to process the message, upon successful process, it sends an acknowledgement to the Q and the message gets deleted from the Q. if unsuccessful, after a defined period if the Q doesnt get an acknowledgement, the message is appended at the end of the Q . In this way we dont loose any message.
With my little knowledge in Kafka I understand that if for example message 100 was not successfully processed, the offset was not increased, but it will be increased if message 101 is processed successfully . So I lost the message 100.
Is there a way to guarantee that none of the messages will be lost .
I also faced the same question. If I want to put in a simple way, RabbitMQ keeps a count of each
Kafka doesnt, so u cant have it ready made, you have to implement it urself.
There are options available though, use kmq, performance will become less than 50% , have a look
https://softwaremill.com/kafka-with-selective-acknowledgments-performance/