Search code examples
apache-kafkaudpfluentd

Filtering in Fluentd issue


I'm receiving data through a loadbalancer to my Kafka cluster (3 nodes). Besides nginx logmessages that I need, the loadbalancer is also sending me sort of pings to see if the kafka nodes are still alive. These 'pings' appear in my Kafka topic as :

{"message":"A10\u0000"}

I've been trying to filter out these messages in the fluentd config file, but it seems I can't get it right.

This is (part of) my fluentd config :

<source>
  @type udp
  <parse>
    @type none
  </parse>
  port 11514
  bind 0.0.0.0
  tag trb-login
</source>

<filter A10>
  @type grep
  @log_level debug
  <exclude>
    key message
    pattern A10
  </exclude>
</filter>

Also tried several other regexps for pattern, like ^A10, ^A10$, ^A10*. Even tried the full text for pattern (A10\u0000), with various numbers of backslashes. Nothing lets me filter out this line... (by the way: the @log_level debug does nothing either....)

My version of fluentd is 1.2.2

Am I using the right vehicle here (filter) ? Or should I use parser ? Any ideas ?


Solution

  • Try the following configuration.

      <filter>
        @type grep
        <exclude>
          key message
          pattern /^A10/
        </exclude>
      </filter>