Using flume to import data from kafka topic to hdfs folder

Question

I am using flume to load messages from kafka topic HDFS folder. So,

1. I created a topic TT
2. I sent messages to TT with a kafka console producer
3. I configured the flume agent FF
4. Run the flume agent flume-ng agent -n FF -c conf -f flume.conf - Dflume.root.logger=INFO,console

The Code Execution Stops, without Error and it doesnot write anything to HDFS. The log file contains this warning No broker partitions consumed by consumer thread flume_-0 for topic.

Any help is greatly appreciated. Thanks in advance.:)

Answer 1

this conf fixed the problem

• FF.sources=kafka-source
• FF.channels = kafka-channel
• FF.sinks =hdfs-sink FF.channels.kafka-channel.type org.apache.flume.channel.kafka.KafkaChannel
• FF.channels.kafka-channel.kafka.bootstrap.servers = wided-Latitude:9092
• FF.channels.kafka-channel.kafka.topic = TT
• FF.channels.kafka-channel.parseAsFlumeEvent = false
• FF.sinks.hdfs-sink.hdfs.writeFormat = Text
• FF.sinks.hdfs-sink.channel = kafka-channel
• FF.sinks.hdfs-sink.type = hdfs
• FF.sinks.hdfs-sink.hdfs.path = hdfs://***:8020/spark/