Search code examples
hadoopbigdataroutersyslogflume

flume for collecting syslog data


I am trying to collect syslog from 10 devices(routers). I came to know that I can use syslog source, but need clarification about the host and ports in the properties. Whether they are the local port on the machine where flume agent is running. Also how to redirect syslogs to the port which flume is listening to.


Solution

  • The classic Syslog source is basically designed to connect to one syslog host, i.e. you will have to set up 10 sources for your 10 syslog servers. All those sources can run in one agent and spool their events to one sink using one channel - this setup will however soon run into performance issues if the amount of data is significant. You will have to configure your routers to connect to that syslog host/port config.

    The larger setup would be to install one agent for one syslog server each and have that spool the events using Avro sink / Avro Source to one or two agents which then again sort the events and write them where your want to have them.

    You can also use the newer Multiport Syslog TCP Source which spawns several ports. This allows more routers to connect to one syslog source:

    a1.sources = r1
    a1.channels = c1
    a1.sources.r1.type = multiport_syslogtcp
    a1.sources.r1.channels = c1
    a1.sources.r1.host = 0.0.0.0
    a1.sources.r1.ports = 10001 10002 10003
    a1.sources.r1.portHeader = port
    

    in this config, syslog offers the ports 10001,10002 and 10003 on your agent. You would configure your routers to connect to your agent's IP and one of these three ports.