Search code examples
apache-kafkakafka-consumer-apikafka-producer-api

Kafka - difference between Log end offset(LEO) vs High Watermark(HW)


What is the difference between LEO and HW in Replica ( Leader Replica)?

Will they contain the same number? I can understand HW is the last committed message offset.

When LEO will be updated and how?


Solution

  • The high watermark indicates the offset of messages that are fully replicated, while the end-of-log offset might be larger if there are newly appended records to the leader partition which are not replicated yet.

    Consumers can only consume messages up to the high watermark.

    See this blog post for more details: http://www.confluent.io/blog/hands-free-kafka-replication-a-lesson-in-operational-simplicity/