Search code examples
apache-kafkaconsumer

kafka : How to delete data which already been consumed by consumer?

I set server.properties'

log.retention.minutes=8

to clean data under kafka-logs/ every 8 minutes automatically ,

is it possible let the cleaner only clean up the data which have been consumed ,data not consumed by consumer will retain ?

Thanks !

Solution

  • No. Kafka messages are appended to log files which roll over every x hours or when they reach a certain size (depending on configuration). Once rolled over, those files are immutable (you cannot delete individual records). Log files are cleaned up when the last write access to a file exceeds the retention time.

    In other words: the retention time is the time a message is kept at least. It is possible for a message with retention time of minutes to last for weeks (depending on other configuration settings).

    The concept of "consumer offsets" is the mechanism Kafka uses to avoid reconsumption of messags. Kafka 0.11 also will contain exactly-once capabilities.