Search code examples
hdfsflumedata-stream

creating flume agent in hortonworks sandbox vm for streaming weather data into HDFS


I am configuring flume to stream weather data, i have written flume agent as given below,

WeatherAgent.sources= Weather

WeatherAgent.sinks = hdfs-write

WeatherAgent.channels= memory-channel

WeatherAgent.sources.Weather.type = weather

WeatherAgent.sources.Weather.bind = api.openweathermap.org/data/2.5/forecast/city?id=524901&APPID=*********

WeatherAgent.sources.Weather.port = 11111

WeatherAgent.sinks.hdfs-write.type = hdfs

WeatherAgent.sinks.hdfs-write.hdfs.path = hdfs://localhost:8020/user/hadoop/flume

WeatherAgent.sinks.hdfs-write.rollInterval = 30

WeatherAgent.sinks.hdfs-write.hdfs.writeFormat=Text

WeatherAgent.sinks.hdfs-write.hdfs.fileType=DataStream

WeatherAgent.channels.memoryChannel.type = memory

WeatherAgent.channels.memoryChannel.capacity=10000

WeatherAgent.sources.Weather.channels=memoryChannel

WeatherAgent.sinks.hdfs-write.channel=memoryChannel

i am getting error as

weather agent does not contain any valid channels making it as invalid.

i am new to flume,i dont know which all parameters to be addded in agent config file,i even dont know written agent is correct or wrong,please help me to solve this. if any Help links are there to write custom or new agents please post.

thanks in advance

Suyog


Solution

  •  it is type mismatching. In your configuration file you mentioned  
    
        WeatherAgent.channels= memory-channel
    
      but you are using  memoryChannel instead of memory-channel
    
        WeatherAgent.channels.memoryChannel.type = memory
    
        WeatherAgent.channels.memoryChannel.capacity=10000
    
        WeatherAgent.sources.Weather.channels=memoryChannel