Search code examples
apache-kafkaapache-nifi

On NiFi how to differentiate messages coming from two different Kafka?

I have two different applications named A1 and A2. Each application has its own Kafka server. Messages coming from these two KAFKA servers (brokers) goes to NiFi.

Each Kafka has different topic names and based on that I can differentiate messages coming from Kafka. But apart from Kafka's topic name are there any other ways in NiFi to differentiate messages coming from two different Kafka? Are there any NiFi processor which checks the topic name and then decide the next route to take?

Solution

  • If you're using NiFi's Kafka processors (ConsumeKafka/ConsumeKafkaRecord) to receive messages from Kafka, they will output the messages in the form of FlowFiles. They carry one attribute called kafka.topic which will have the name of the topic the message came from.

    To route the messages based on the topic name, you can use RouteOnAttribute processor. Say for example, you have two topics topicA & topicB. Then you have to configure the RouteOnAttribute processor like this:

    enter image description here

    enter image description here

    Then connect the relationship topic-a & topic-b to separate flows based on your requirement. If you're adding more Kafka sources, all you have to do is, update RouteOnAttribute with one more dynamic relationship. Ex: topic-c : ${kafka.topic:equals('topicC')}