Creating custom grok filters

Question

I need to find grok pattern for files where the lines are of the format :

 3 dbm.kfa            0        340220       7766754          93.9
 3 swapper/3            0        340220       7766754          93.9

This is the grok pattern that I have done so far.

\s*%{INT:no1}\s*%{USERNAME:name}\s*%{INT:no2}\s*%{INT:no3}\s*%{INT:no4}\s*%{GREEDYDATA:ans}

The field USERNAME works for dbm.kfa but not for swapper/3 as USERNAME does not include \ character. I would like to create some custom filter for this purpose, but have no idea how to create one.

Any help would be really appreciated. Thanks a lot !

Answer 1

To create a custom pattern you need to use an external file in the following format and put that file in a directory the will be used only for pattern files.

PATTERN_NAME [regex for your pattern]

Then you will need to change your grok config to point to the pattern files directory.

grok {
    patterns_dir => ["/path/to/patterns/dir"]
    match => { "message" => "%{PATTERN_NAME:fieldName}" } 
}

But in your specific case if you change %{USERNAME:name} to %{DATA:name} it should work.

For a better explanation about the custom patterns you should read this part of the documentation.

You also can find all the core grok patterns that ships with logstash in this github repository, the most used are in the grok-patterns file.