Search code examples
apache-kafkaapache-kafka-streams

Standby tasks not writing updates to .checkpoint files

I have a Kafka Streams application that is configured to have 1 standby replica created for each task. I have two instances of the application running. When the application starts the application writes .checkpoint files for each of the partitions it is responsible for. It writes these files for partitions owned by both active and standby tasks.

When sending a new Kafka event to be processed by the application, the instance containing that active task for the partition updates the offsets in the .checkpoint file. However, the .checkpoint file for the standby task on the second instance is never updated. It remains at the old offset.

I believe this is causing us to see OffsetOutOfRangeEceptions to be thrown when we rebalance which results in tasks being torn down and created from scratch.

  1. Am I right in thinking that offsets should be written for partitions in both standby and active tasks?
  2. Is this an indication that my standby tasks are not consuming or could it be that it is purely not able to write the offset?
  3. Any ideas what could be causing this behaviour?

Streams version: 2.3.1

Solution

  • This issue has been fixed in Kafka 2.4.0 which resolves the following bug issues.apache.org/jira/browse/KAFKA-8755

    Note: The issue looks to only effect applications the are configured OPTIMIZE="all"