Search code examples
apache-kafkakafka-topic

Kafka: What happens to message with null key when log compaction is enabled?

What happens to message with null key when log compaction is enabled? Are messages with a null key compacted?

Solution

  • In current version what happens is this:

    "Currently when a user sends a null key to a log compacted topic, the broker returns CorruptRecordException, which is a retriable exception. As such, the producer keeps retrying until retries are exhausted or request.timeout.ms expires and eventually throws a TimeoutException."

    As reported in Jira KIP-135 this is a bit confusing.

    In the future if the cleaner solution from KIP-135 (still under discussion as @Ran Lupvoich pointed out) get accepted another behavior should be the following:

    When you try to produce null key to log compacted topic, Kafka should return a new error type INVALID_KEY