Search code examples
apache-kafka-streamsspring-cloud-stream

Sprint Cloud Stream Kafka Streams Binder processor application stuck

I have the following Spring Cloud Stream Kafka Streams Binder 3.x application:

enter image description here

When I run X messages through this application by publishing them to the topic1 from an integration test using @SpringBootTest and @EmbeddedKafka the counts of messages at points 1 and 2 are equal, as I expect.

When I do the same using live application connected to the Kafka broker, the counts at point 1 and point 2 remain significantly different: Count1 >> Count2.

Kafka Tool shows a big Lag of the Processor2 consumer on the topic2 and that lag remains constant (doesn't change after I stop publishing messages)

The Processor2 consists of

  • flatTransform stateful transformer
  • aggregator
  • other downstream steps

What could be the reason of the distinct behaviour during test and live mode and Lag not going down in live mode?

I have thoroughly compared all application property values active in test and in live application, they are exactly equivalent.

There is only 1 partition in all topics in both cases.

Solution

  • In my case the reason was default 7 days retention setting of the topics that were automatically created by the Spring Cloud Stream application.

    The messages in my input stream span 8 years, I am using custom TimestampExtractor.

    After I have manually configured topics to a large retention time, the issue was solved:

    /usr/bin/kafka-configs --bootstrap-server localhost:9092 --alter --entity-type topics --entity-name topic2 --add-config retention.hours=87600

    Or set the log.retention.hours for the entire Kafka broker.