Search code examples
apache-kafkakafka-topickafka-partition

Does Kafka chose an alternative partition in the same topic if the partition I want to send message to was full of disk?


My topic has 3 partitions (in 3 different brokers). I want to send message to particular partition 1 (add partition id or specify a key in my message). Assuming the partition 1 became full of disk, does an alternative partition (partition 2 or 3 in this case) be chosen for this message?


Solution

  • It is not the topic that is running out of disk but only the entire broker. If the broker runs out of disk it will shut itself down and you will not be able to send to or receive data from it.

    As you are specifically pointing to a partition, it now depends if you have a replication-factor value larger than 1. In case you have replicated your topic partitions, the partition leader would switch from the dead broker to another broker with that has a partition in-sync.

    If you do not have replication of more than 1 your producer would end up getting an OutOfMemory error and the data it not produced to the alternative partitions 2 or 3.

    To make sure that this never happens you can apply the volume based retention policies that is available in Kafka if you set your cleanup.policy to delete and set the retention.bytes such that with the number of topics/replicas you will not exceed your storage limit.