Search code examples
apache-kafkakafka-producer-api

Choose number of partitions in a topic?

How do I go about the process of choosing number of partition for a topic?

What happens if records are going to partitions according to key and there are more keys than no of partitions?

Is there only one way that records with same keys go to one partition? Can't a developer customize it?

Solution

    • you can specify number of partitions when you create topic. Later you can increase number of partitions

    • if you use default partitioner class in kafka, all records with the same key will go to one partitions.

      return Utils.abs(Utils.murmur2(record.key())) % numPartitions;

    • you can specify your own partitioner class and implement any logic you want inside. For that you need to override producer property partitioner.class with your own implementation.