Search code examples
flumeflume-ng

Duplicate channel before being intercepted by interceptor

I'm using flume to do something like this

Source --> interceptor --> Channel --> multiplexing --> HDFS Sink
                                          |-----------> Null Sink

I would like to add a channel just after the source but I don't want event pass through the interceptor. I would like "raw" event. Like this:

Source --> interceptor (i) --> Channel --> multiplexing --> HDFS Sink
    |                                     |-----------> Null Sink
    |-------> Channel (must no be intercepted by i) --> HDFS

How can I do it ?

Thanks

Solution

  • Since interceptors are configured per source, you will have to add a second source (configured with no interceptors at all and listening in a different Http port), and emit your data twice: one copy for the source with interceptors, and one copy to the other source.

    Another possibility is to chain two agents. The first one containing a single source with no interceptors, and two sinks: one for persisting the data as it is in HDFS, and the other feeding the agent you already have. I mean:

    src-->ch-->multip-->sink----------->src-->int-->ch-->multip-->hdfssink
                |-->hdfssink                              |-->nullsink
(________agent1____________)        (_____________agent2_____________)