Flafka : How to write into partitoined Kafka topic (channels) which is partitioned on msg type through Flume agent

Question

In my project , We have total 11 WSMQ as source for Flume agents. Kafka topic as channel, Kafka Topic is partitioned on message types. HDFS as sink.

Requirement: We want to read from multiples queues using flume-agent and write to specific partition of kafka topic so that at later point of time , we could read data from those kafka partitions.

Right now, we have total 11 kafka topics for each queue where agents are writing the messages. We want to have a single Kafka topic which is partitioned on msg types and incoming msgs could be written to those partitions.

Can anybody suggest , what would be best approach for this use case?

Thanks!

Answer 1

We solved it by Spark streaming.