How to separate topic partitions into multiple kafka servers?

Question

I want to separate topic partitions into multiple kafka servers like on this picture.

Example: I have only one topic "Test", I have two kafka servers (k1, k2), I have 4 partitions of topic "Test". I want to server k1 have partitions 0 and 1, and server k2 have partitions 2, 3.

Answer 1

Kafka servers are nothing but multiple brokers. Please follow below steps,

1. Replicate the config/server.properties file into the number of servers you need. In your case its two. Copy the file to two different file and change the broker id and port

   i. server_K1.properties (you can provide the file name as you wish), change the file properties as below.
   broker.id=1
port=9092
log.dir=/tmp/kafka-logs-1
   

   ii. server_K2.properties
broker.id=2
port=9093
log.dir=/tmp/kafka-logs-2
   

Start the kafka server with above two configurations

bin/kafka-server-start.sh config/server_K1.properties
bin/kafka-server-start.sh config/server_K2.properties

By providing the replication factors the Topics will be replicated across the brokers. And based on the partitions you create, and the amount of data you receive, the partitions are shared across the brokers.

You can create 4 partitions while creating the Topic and this partitions will be shared across the brokers.