Search code examples
flumecloudera-manager

How to run multiple Flume agents on a single node with Cloudera Manager?


I have installed Flume on my CDH5.8.0 cluster. The flume agent is on a single node where Flume tasks run. I use the linux terminal ro run two sepate data ingestions via flume, with seperate configuartion files.

I want to monitor both ingestion processes via Cloudera Manager. Although the flume configuration panel in CM gives the option to add custom conf properties, but that is for a single flume agent.

I cant add another flume agent via CM on the same host. How can I monitor both ingestion processes with Cloudera Manager?


Solution

  • If you can only monitor a single Flume agent, I think you could merge both agent configuration files in a single one, in order to run a single Flume agent (the one that you could monitor).

    You can declare as many sources, channels and sinks as you want:

    a1.sources = r1 r2
    a1.sinks = k1 k2
    a1.channels = c1 c2
    

    And then bind them appropriately:

    a1.sources.r1.channels = c1
    a1.sources.r2.channels = c2
    ...
    a1.sinks.k1.channel = c1
    a1.sinks.k2.channel = c2