Search code examples
apache-kafka-streams

KStreams - org.apache.kafka.common.errors.TimeoutException: Timeout of 60000ms expired before successfully committing offsets


On kafka Streams(version: 2.3.1), we are facing issues with committing offsets:

org.apache.kafka.common.errors.TimeoutException: Timeout of 60000ms expired before successfully committing offsets

This is not happening regularly. What could be the reason?

Also, since TimeoutException is a retriable exception, we were planning to increase retries if it's something intermittent error.

Will it help in anyway? We have ATLEAST_ONCE processing guarantee and ordering matters in our use-case.

We hope it won't impact offsets order in any way, since KStreams commits offsets in sync, and if one commit fails and is being retried, the particular stream thread will not process new records and offsets would not be disturbed.


Solution

  • If the TimeoutException is transient it could be stemming from intermitted network issues or an overload of some brokers, in which case, increasing the retries may help.

    However it might be better in terms of latency to discover the root cause of the timeout. Inorder to find to root cause you might want to look into the meterics more. Here is a blog that will get you started https://sematext.com/blog/kafka-metrics-to-monitor/

    It might be fixable by giving the bottleneck more resources.