Search code examples
apache-kafkaflumeflume-ng

Is it possible to use Flume Kafka Source by itself?


Let's say there are a bunch of producers writing avro records (that have the same schema) into a Kafak Topic.

Can I use Flume Kafka Source to read those records and write them to HDFS. Even though the records were not published using a Flume Sink?


Solution

  • Yes, you can. In general, what a Kafka consumer can or cannot do is fully independent on who produced the data, but only on the format in which the data has been encoded.

    That is the whole point of Kafka and Enterprise Service Bus.