Search code examples
hadoopapache-kafkaflume

Writing to custom HDFS directory based on message data for Kafka -> Flume -> hdfs ingestion

How can I write to custom hdfs directory using flume based on message type in Kafka message ?

Say kafka message: {"type": "A", "data": "blah"} having "A" in type field should be written to /data/A, message: {"type": "B", "data": "blah"} having "B" in type field should be written to /data/B, etc

Solution

  • I think you need Custom flume sink. Apache Flume custom sink