Search code examples
apache-kafkakafka-topicapache-kafka-mirrormaker

Create Kafka topic in Active-Passive Kafka cluster architecture

I'm trying to figure out what is the proper way to create a topic in Active-Passive Kafka cluster architecture.

Let's say we have two Kafka clusters (Active-Passive) in two separated availability zone. I understand that the topic mirroring happens when data is replicated from active to passive clusters.

The question is. When I create a topic should I create it on both clusters separately? If not, then the Mirroring maker should know to detect if there is a new topic on the Active cluster and create its replica on the passive cluster.

Will appreciate any clarification

Solution

  • KMM2 (at least) does monitoring of Kafka topics in the source cluster.

    In detail take a look at how often KM does use the admin to find out what are the topics to replicate and how it is involved in syncing their configuration.

    KMM code is pretty straightforward, so forking and making changes if needed should not be too hard (but it is very much possible you might get what you want by setting up policies/params).