I'm receiving data through a loadbalancer to my Kafka cluster (3 nodes). Besides nginx logmessages that I need, the loadbalancer is also sending me sort of pings to see if the kafka nodes are still alive. These 'pings' appear in my Kafka topic as :
{"message":"A10\u0000"}
I've been trying to filter out these messages in the fluentd config file, but it seems I can't get it right.
This is (part of) my fluentd config :
<source>
@type udp
<parse>
@type none
</parse>
port 11514
bind 0.0.0.0
tag trb-login
</source>
<filter A10>
@type grep
@log_level debug
<exclude>
key message
pattern A10
</exclude>
</filter>
Also tried several other regexps for pattern, like ^A10, ^A10$, ^A10*. Even tried the full text for pattern (A10\u0000), with various numbers of backslashes. Nothing lets me filter out this line... (by the way: the @log_level debug does nothing either....)
My version of fluentd is 1.2.2
Am I using the right vehicle here (filter) ? Or should I use parser ? Any ideas ?
Try the following configuration.
<filter>
@type grep
<exclude>
key message
pattern /^A10/
</exclude>
</filter>