I have a task to use Apache Flume to send messages to a Kafka topic. The caveat is that I have to specify a partition based on an IP address that will be in the message.
Is there a way to configure Apache Flume to do this dynamically, or do I have to implement a custom Producer plugin?
Thank you.
After some research, the only real custom way of configuring the producer is to roll your own.
I implemented the AbstractSink interface and put in my own hash algorithm to generate a partition to send the message to.