Search code examples
logstashlogstash-grok

what will be the Grok pattern for this custom log pattern?

Following is a small part of my log :

2018-12-06 18:55:20 INFO  epo - myfile.xml is loaded successfully
2018-12-06 18:55:20 INFO  epo - checking that whether the given file name is already present
2018-12-06 18:55:20 INFO  epo - some logging deatils
2018-12-06 18:55:20 INFO  epo - Entry has been added to table.
2018-12-06 18:55:20 INFO  epo - Total number of records processed 0000035
2018-12-06 18:55:20 INFO  epo - some logging deatils
2018-12-07 09:57:59 INFO  epo - myfile.xml is loaded successfully
2018-12-07 09:57:59 INFO  epo - [ElasticSearch] => PIN07122018F00001 request sent successfully.
2018-12-06 18:55:20 INFO  epo - myfile.xml is loaded successfully
2018-12-06 18:55:20 INFO  epo - checking that whether the given file name is already present
2018-12-06 18:55:20 INFO  epo - some logging deatils
2018-12-06 18:55:20 INFO  epo - Entry has been added to table.
2018-12-06 18:55:20 INFO  epo - Total number of records processed 0000035
2018-12-06 18:55:20 INFO  epo - some logging deatils
2018-12-07 09:57:59 INFO  epo - myfile.xml is loaded successfully
2018-12-07 09:57:59 INFO  epo - [ElasticSearch] => PIN07122018F00002 request sent unsuccessfully.

In this log I want to select lines which contains request IDs like PIN07122018F00001 and PIN07122018F00002 and send it to elastic Search.

I am using logstash for this purpose, and my grok pattern is :

input {
  . . .
}

filter {
  grok {
    patterns_dir => ["/myServer/mnt/appln/folder1/folder2/logstash/pattern"]
    match => { "message" => '^%{TIMESTAMP_ISO8601:timestamp} INFO  epo - \[ElasticSearch\] => %{REQ_ID:requestid} %{MSG:statusmsg}$' }
  }
}

output{
    . . .
}

where DEPOSITORY_REQ_ID and MSG is defined as : 

MSG (A-Za-z0-9 )+
REQ_ID PIN[0-9]{8}[A-Z]{1}[0-9]{5}

But I am still not able to match the required line, with this pattern its taking all the lines. Please tell me what will be the pattern to match the line :

2018-12-07 09:57:59 INFO epo - [ElasticSearch] => PIN07122018F00001 request sent successfully.

Please Help.

Solution

  • The issue is with the MSG pattern. The () denote a capturing group, which will try to match the exact content of the (). What you want to use in your case is [], which denotes a character class, which will match all characters from that class. Also it's missing the . that appears at the end of the lines.

    Your pattern should be defined this way, which would fix your issue:

    MSG [A-Za-z0-9 \.]+