Search code examples
flumeflume-ng

Flume: are Http handlers spawned into parallel threads?


A Http source in Flume depends on a handler in charge of creating Event objects to be put in the channel(s). Does this handler spawn into several threads, let's say one per incoming http message? If the answer is "yes", is there any kind of pool? Which is the size of such a pool? If the answer is "no", then must I assume the incoming http messages are processed sequentially? (I hope they are not, and some kind of parallelism is implemented).

Thanks!


Solution

  • By looking at https://github.com/apache/flume/blob/trunk/flume-ng-core/src/main/java/org/apache/flume/source/http/BLOBHandler.java#L63 for example (and https://github.com/apache/flume/blob/trunk/flume-ng-core/src/main/java/org/apache/flume/source/http/HTTPSource.java#L241 ), it looks single-threaded to me.

    You can parallelize by configuring multiple HTTP Sources.

    Also, the "pool" size would be defined by the channel you connect the source to, if I understand your question correctly.