Write a custom HTTPSource Handler for Flume-ng

Are there any resources available to help me out in writing a custom handler for a HTTPSource for Flume-ng. I read the documentation and there is a sample handler for Json but I am wondering if anybody has had the need to write a handler for creating Flume events from a XML message body. The HttpSource is now available in Flume-ng 1.3.1 but we need handlers to interpret our data.

Thanks.

Solution

Did you look at JSONHandler source? The only difference for XMLHandler would be usage of some XML deserializer instead of Gson.

You just need to convert something like:

<dataList>
  <data>
     ...
  </data>
  <data>
     ...
  </data>      
</dataList>

into List<SimpleEvent> (or introduce your own HTTPEvent, as Flume developers did with JSONEvent, if you need to handle different encodings). These events' bodies will contain your <data>..</data> chunks as byte representation of String.

I don't recommend to implement any additional business logic of parsing these events in Flume, because business rules tends to change often and Flume as infrastructure software should be stable.

Finally, you pack your code into jar, place this jar into Flume's lib directory (/usr/lib/flume-ng/lib in case of Cloudera's rpm distribution), specify in flume.conf "handler" property of HTTPSource as "com.vicky.flume.source.http.XMLHandler" (or something like that), restart the agent and that's all.

Hope this helps. Will be glad to answer your questions.