Search code examples
logstashlogstash-grok

Logstash Grok Filter - parsing custom file

I am finding that logstash is not a fan of my filter. Would be nice to have a second set of eyes on it.

First - my log file - has the following entries with new lines for every volume.

/vol/vol0/ 298844160 6916836 291927324 2%  /vol/vol0/

My config file looks as follows:

INPUT

file {
   type => "testing"
   path => "/opt/log_repo/ssh/netapp/*"
   tags => "netapp"
   start_position => "beginning"
   sincedb_path => "/dev/null"
}

FILTER

if [type] == "testing" {
   grok{
        match => [ "@message", "{UNIXPATH:volume}%{SPACE}%{INT:total}%{SPACE}%{INT:used}%{SPACE}%{INT:avail}%{SPACE}%{PROG:cap}%{SPACE}%{UNIXPATH:vols}"]
   }
}

OUTPUT

if [type] == "testing" {
     elasticsearch {
         action => "index"
         hosts => ["http://localhost:9200"]
         index => ["testing4-%{+YYYY.MM.dd}"]
     }
}

When I run this it tells me it has a bad config file. If I change the filter to: 

match => [ "@message", "{UNIXPATH:volume}" ]

It creates a new field called volume with the volume name. I am using the space component because the log is simply not consistent. Some volumes will have 4 spaces between the usable space and some will have more or less depending on the volume name and the size.

To get to this configuration i leveraged the following sites: https://grokdebug.herokuapp.com/discover?# http://grokconstructor.appspot.com/do/constructionstep

Still struggling on what I am missing.... Any help would be greatly appreciated.

UPDATE: After adding the recommendation below it still doesn't create a new field. 

_index      string              
message     string              
type    string              
tags    string              
path    string              
@timestamp      date                
@version    string              
host    string              
_source     _source             
_id     string              
_type   string              
_score
Solution

  • Your pattern doesn't matrch the sample log for a very simple and silly reason - you are missing % at the start of your pattern. If you will add it then it works like a charm:

    enter image description here

    So the full filter is:

    if [type] == "testing" {
   grok{
        match => [ "@message", "%{UNIXPATH:volume}%{SPACE}%{INT:total}%{SPACE}%{INT:used}%{SPACE}%{INT:avail}%{SPACE}%{PROG:cap}%{SPACE}%{UNIXPATH:vols}"]
   }
}