Search code examples
apache-kafkadistributed-system

how does kafka synchronize data timestamps between different brokers and how is timestamp created in kafka


How does Apache Kafka manage synchronizing timestamps between different brokers in a cluster, which time is actually recorded and converted as the message creation timestamp and how does Kafka verify it's time precision


Solution

  • The record timestamp is stored as part of the record's bytes. The bytes of each record are copied between brokers, including the timestamp.

    By default (depending on the client), the producer client generates/sets the timestamp of the record, not the broker

    You otherwise might want to look at the broker properties for message timestamps

    log.message.timestamp.difference.max.ms

    The maximum difference allowed between the timestamp when a broker receives a message and the timestamp specified in the message. If log.message.timestamp.type=CreateTime, a message will be rejected if the difference in timestamp exceeds this threshold. This configuration is ignored if log.message.timestamp.type=LogAppendTime

    Type: long
    Default: 9223372036854775807