Search code examples
javaapache-kafkakafka-producer-apitimeoutexception

Guidelines to handle Timeout exception for Kafka Producer?

I often get Timeout exceptions due to various reasons in my Kafka producer. I am using all the default values for producer config currently.

I have seen following Timeout exceptions:

org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms.

org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s) for topic-1-0: 30001 ms has passed since last append

I have following questions:

  1. What are the general causes of these Timeout exceptions?

    1. Temporary network issue
    2. Server issue? if yes then what kind of server issue?

  2. what are the general guidelines to handling the Timeout exception?

    1. Set 'retries' config so that Kafka API does the retries?
    2. Increase 'request.timeout.ms' or 'max.block.ms' ?
    3. Catch the exception and have application layer retry sending the message but this seems hard with Async send as messages will then be sent out of order?

  3. Are Timeout exceptions retriable exceptions and is it safe to retry them?

I am using Kafka v2.1.0 and Java 11.

Thanks in advance.

Solution

  • The default Kafka config values, both for producers and brokers, are conservative enough that, under general circumstances, you shouldn't run into any timeouts. Those problems typically point to a flaky/lossy network between the producer and the brokers.

    The exception you're getting, Failed to update metadata, usually means one of the brokers is not reachable by the producer, and the effect is that it cannot get the metadata.

    For your second question, Kafka will automatically retry to send messages that were not fully ack'ed by the brokers. It's up to you if you want to catch and retry when you get a timeout on the application side, but if you're hitting 1+ min timeouts, retrying is probably not going to make much of a difference. You're going to have to figure out the underlying network/reachability problems with the brokers anyway.

    In my experience, usually the network problems are:

    • Port 9092 is blocked by a firewall, either on the producer side or on the broker side, or somewhere in the middle (try nc -z broker-ip 9092 from the server running the producer)
    • DNS resolution is broken, so even though the port is open, the producer cannot resolve to an IP address.