How to configure spring-kafka's partition and replication in distributed environment?

Question

I want to configure 3 partitions and 3 replications of a topic in distributed environment with three nodes. How can I configure these by java api without shell command?

If I have three nodes: node1, node2 and node3. I want partition1 and replication3 are deployed in node1, partition2 and replication1 are deployed in node2, partition3 and replication2 are deployed in node3.

I've tried spring-kafka's api in single-machine environment, this can create a topic and 1 partition automatically. But it not work in distributed environment.

My maven configuration is:

<dependency>
    <groupId>org.springframework.kafka</groupId>
    <artifactId>spring-kafka</artifactId>
    <version>1.1.7.RELEASE</version>
</dependency>

Answer 1

1.1.x is no longer supported; you should be using at least 1.3.9.

1.3.x comes with KafkaAdmin, which can automatically configure any NewTopic beans in the application context.

See Configuring Topics.

If you define a KafkaAdmin bean in your application context, it can automatically add topics to the broker. Simply add a NewTopic @Bean for each topic to the application context.

@Bean
public KafkaAdmin admin() {
    Map<String, Object> configs = new HashMap<>();
    configs.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG,
            StringUtils.arrayToCommaDelimitedString(kafkaEmbedded().getBrokerAddresses()));
    return new KafkaAdmin(configs);
}

@Bean
public NewTopic topic1() {
    return new NewTopic("foo", 10, (short) 2);
}

@Bean
public NewTopic topic2() {
    return new NewTopic("bar", 10, (short) 2);
}