Search code examples
nginxflumesyslogrsyslog

Flume. Strip all tags from syslogudp source

I am using apache-flume with syslogudp source. I set following into flume.conf:

agent.sources.r1.type = syslogudp
agent.sources.r1.port = 5140
agent.sources.r1.host = localhost
agent.sources.r1.channels = c1

And following nginx.conf options:

access_log syslog:server=localhost,tag= ;

As result all my logs in sink (kafka topic) have 2 symbols in beginning. For example:

: 31.130.95.75  1472493421.911  80674   {flags}

Is there any way to cut off first colon and space symbols?

thanks

Solution

  • You might want to take a look at the Search and Replace Interceptor ( https://flume.apache.org/FlumeUserGuide.html#search-and-replace-interceptor )

    You can use the interceptor to edit event bodies using regular expressions.

    In your case, the modified config would look like this:

    agent.sources.r1.type = syslogudp
agent.sources.r1.port = 5140
agent.sources.r1.host = localhost
agent.sources.r1.channels = c1


agent.sources.r1.interceptors = search-replace
agent.sources.r1.interceptors.search-replace.type = search_replace

agent.sources.r1.interceptors.search-replace.searchPattern = ^.*:
agent.sources.r1.interceptors.search-replace.replaceString =