How to parse log file with different types of messages

Question

I have a log file which contains complicated message types. Here is an example:

2016-07-07 13:30:02 [Main] *** Program start ***
2016-07-07 13:30:02 [UnzipFile] Before file collection
2016-07-07 13:30:02 [GetZipCol] Start get sorted zip file collection
2016-07-07 13:30:02 [GetZipCol] End get sorted zip file collection
2016-07-07 13:30:02 [Main] [ERROR] No unzip file
2016-07-07 13:30:03 [Main] *** Program end ***

The following grok pattern is only suitable for first 4 lines but not the 5th line.

grok{
    match => {"message" => ['%{Date:Date}%{SPACE}%{Time:Time}%{SPACE}%{WORD:Job}%{SPACE}%{GREEDYDATA:Message}']}
        }

I would like to know how should I modify the grok pattern as to capture[ERROR] from the last message. Is there anyone know how the way to do this?

This is my output part in conf

if [Message] == "*** Program start ***" {
    elasticsearch { 
    hosts => ["localhost:9200"] 
    index => "log-%{+YYYY.MM.dd}"
    template => "C:/logstash/log.json"
    template_overwrite => true
    }   
}
if [Message] == "*** Program end ***" {
    elasticsearch { 
    hosts => ["localhost:9200"] 
    index => "log-%{+YYYY.MM.dd}"
    template => "C:/logstash/log.json"
    template_overwrite => true
    }   
}  
if [Level] =~ /.+/ {
    elasticsearch { 
    hosts => ["localhost:9200"] 
    index => "log-%{+YYYY.MM.dd}"
    template => "C:/logstash/log.json"
    template_overwrite => true
    }   
}

If I only want to grasp the event when the Program starts and ends and also the events with errors while the other events can be dropped. However, according to what I have written. I can only grasp the data with [Error]. How should I also grasp the other data? And will there be a simpler way of doing that instead of typing 3 if conditional statements? Thanks.

Thanks.

Answer 1

You can use two pattern in the same grok filter, if the first one fails, the second is use. So in your case, the first pattern will try to capture the [ERROR], the second will be the pattern from your answer.
I think it's more readable.

grok{
  match => {
    "message" => [
       '%{DATE:Date}%{SPACE}%{TIME:Time}%{SPACE}\[%{WORD:Job}\]%{SPACE}\[%{WORD:Level}\]%{SPACE}%{GREEDYDATA:Message}',
       '%{DATE:Date}%{SPACE}%{TIME:Time}%{SPACE}\[%{WORD:Job}\]%{SPACE}%{GREEDYDATA:Message}'
   ]}
}