We are using Kafka MirrorMaker 1 running in an Azure Container Instance to send records from a kafka cluster to a Microsoft Azure Eventhub. Our topic has 10 partitions and we have manually created a topic with the same name and number of partitions in our Azure EventHub. Unfortunately, the records lose their original partitioning after being replicated by MirrorMaker 1. Is there any way to retain the original partitioning using MirrorMaker 1?
It does not; the records are re-sent via DefaultPartitioner
, specifically because MirrorMaker doesn't guarantee the destination has the same number of partitions as the source. You'd need to write your own MirrorMakerMessageHandler
sub-class to accomodate keeping the partitions the same (refer --message.handler
and --message.handler.args
CLI opts). You could also create the destination topic in this class as well.
Since you have the option to run any container, suggest you switch to running a Kafka Connect cluster with MirrorMaker 2 - https://kafka.apache.org/documentation.html#georeplication