Search code examples
rubypuppeterbflumeflume-ng

Flume - Multiple sinks and channels for single agent

Im trying to set up flume such that each agent can have multiple sinks and eventually multiple channels and sources (for now looking at only multiple channels ). I have a config file which looks like this and a ruby template. I am confused about how to go about and add the functionality to the template file such that each agent can send an event to multiple channels

Solution

  • Inside your flume agent, your source will basically push data to multiple agent. FlumeNG by default support multiplexing the events into different channels. There are two options available for that:

    1. Replicating Source: Replicate all data from your source to all the channels

    2. Selective Pushing: Selectively push some data to your channels on the basis of some conditions

    The syntax for that in flume would be something like this:

    dataplatform.sources.source1.selector.type = replicating (or multiplexing)

    (assuming the agent name is dataplatform)

    You can route the data from your source to multiple channels in this way:

    dataplatform.sources.source1.channels = channel1 channel2 channel3

    For further details on this, please refer to flume-ng user guide : https://flume.apache.org/FlumeUserGuide.html