Search code examples
javaapache-kafka-streams

Retention on Kafka Streams > 2.10


I am joining to KStreams thus I need to set JoinWindows. The joined data has computed Timestamps from CustomTimestampExtractors. The Data could be out of order. I can set the retention with the deprecated JoinWindows.until(long)-method but since its deprecated I am looking for another solution.

I found the windowstore.changelog.additional.retention.ms-property am I supposed to use it ? This would lead to all state stores being retained that long.


Solution

  • As the Documentation states:

    Use JoinWindows#grace(Duration) instead

    IMHO the doc is a bit misleading in contrast to:

    Set the window maintain duration (retention time) in milliseconds. This retention time is a guaranteed lower bound for how long a window will be maintained.

    But be careful I wanted my StateStore to retain as long as possible thus I was using JoinWindows.until(Long.MAX_VALUE) before. Using JoinWindows.grace(Duration.ofMillis(Long.Max_VALUE)) does complain that grace period can't be negative. There for you have to use smaller values.