Search code examples
apache-kafkaapache-kafka-streams

Kafka Streams Create Time vs Log-Append Time


If the Kafka broker is configured with Log-Append Time, will the broker overwrite the message timestamp regardless of the timestamp set by the producer/stream?

From streams-time - Documentation, I get the impression that CreateTime is somewhat unreliable - the timestamp is sometimes only forwarded (e.g. when using context.forward()) and sometimes overwritten (e.g. when using puntuate()). I am therefore wondering if the Log-Append Time takes precedence when setting the message timestamp (i.e. overwrites the timestamp set by the stream)?


Solution

  • Yes, LogAppendTime takes precedence, meaning if the topic is configured to use LogAppendTime, the timestamp set by the Streams application (no matter how it is generated) will be ignored and overwritten with the log append time by the broker when the output record is produced.

    Furthermore, I don't think CreateTime is unreliable in the way you think. punctuate() doesn't overwrite an existing timestamp - it is used to generate a fresh record at a specific point in time, so there is no previous CreateTime available - Streams must pick a new timestamp.