Search code examples
logstashlogstash-grok

Grok pattern for different types of log in a logfile

I am trying to write grok pattern for my log file which has three different types of logs, I want to put a filter on the type names (TYPE1,TYPE2,TYPE3) and then write three different grok patterns for this one log file. Also, my log file is a csv separated file.

Log file:
TYPE1,word,word,word,num
TYPE2,word,word,word,word
TYPE3,num,word,num,word

Here's what I have tried so far:

filter {
if [message] =~ /TYPE1/ {
grok {
    match => [ "message", "%{WORD:type},%{WORD:a1"},%{WORD:a2"},%{WORD:a3"},%{POSINT:a4"}]
     }
   }
}

This doesn't work. Also, in this config file i have written grok patterns for other files (which are working well) like: 

filter {
    if [type] == "sometype1" or [type] == "sometype2" {
    grok {
    match => [ "message",  "%{POSINT:moduleid}%{SPACE}%{NUMBER:date}"]
         }
      }
   }

And the logfile which is giving me problem has type=sometype3 which I have not mentioned anywhere.

Thanks

Solution

  • I think you don't need a conditional to do that. If you have static TYPE values ("TYPE1","TYPE2" or "TYPE3") then why not specify one grok pattern for each TYPE:

    filter { 
    grok {
        match => { "message" => [ 
                "TYPE1,%{WORD:a1},%{WORD:a2},%{WORD:a3},%{POSINT:a4}",
                "TYPE2,%{WORD:b1},%{WORD:b2},%{WORD:b3},%{WORD:b4}",
                "TYPE3,%{POSINT:c1},%{WORD:c2},%{POSINT:c3},%{WORD:c4}"  ]
            }
    }
}

    I've tried it and it works for your given formats: 

    TYPE1,word,word,word,num
TYPE2,word,word,word,word
TYPE3,num,word,num,word

    A log file would look like this:

    TYPE1,a,b,c,4
TYPE2,a,b,c,d
TYPE3,1,b,3,d