Search code examples
flumeflume-ng

flume : find ip/hostname of event sender?

I am trying to setup data pipeline where applications servers send (using log4j logging) logevents to flume (using flume log4j appender) over network , to a avrosource that flume agent is using

I tried with below configration but It only appends IP of the host on which agent is running

A1.sources.logE.interceptors = hostint A1.sources.logE.interceptors.hostint.type = org.apache.flume.interceptor.HostInterceptor$Builder A1.sources.logE.interceptors.hostint.preserveExisting = true A11.sources.logE.interceptors.hostint.useIP = false

Solution

  • For anyone who has similar problem , I ended up removing log4j-flume appender at application side , replaced it with log4j-syslog appender. on flume side I configured syslogudp source.

    so finally it looks something like,

    at application side log4j-config 

    log4j.appender.syslog=org.apache.log4j.net.SyslogAppender
log4j.appender.syslog.Facility=LOCAL7
log4j.appender.syslog.FacilityPrinting=false
log4j.appender.syslog.Header=true
log4j.appender.syslog.SyslogHost=flume1.host.net:41473
log4j.appender.syslog.layout=org.apache.log4j.PatternLayout
log4j.appender.syslog.layout.ConversionPattern= DUMMY %p: (%F:%L) %x %m %n

    at flume agent , syslogudp source 

      A1.sources.logMR.type = syslogudp
  A1.sources.logMR.bind = 0.0.0.0
  A1.sources.logMR.port = 41473

    so now my flume event looks like,

    2015-03-17 08:43:02,658 INFO org.apache.flume.sink.LoggerSink: Event: { headers:{timestamp=1426682579754, Severity=7, host=apphost1.host.net, Facility=23, log_type=maplogs} body: 54 4F 46 4C 55 4D 45 20 5B 6D 61 69 6E 2C 5D 20 TOFLUME [main,] }

    and I can now finally acces host key in header with %{host}