I am quite new to logstash so this may be a simple error but I am not able to find where my error is. I have installed logstash and I am trying to parse some logs I generate from a custom java program. I am searching for quite an specific line:
ERROR ProcessStatus 05/24/2017 12:13:58 RETC:0 : Request.evaluate:PDP Response decision: Permit
ERROR ProcessStatus 05/24/2017 12:13:58 RETC:0 : Request.evaluate:PDP Response decision: NotApplicable
I have defined the following config file:
input {
file {
type => "log"
path => [ "/var/log/tomcat7/catalina.out" ]
}
}
filter {
grok {
match => [ "message" , "%{WORD:text1} %{WORD:text2} \[%{DATA:date}\] %{WORD:text3}:%{NUMBER:num1} : %{WORD:text4}.%{WORD:text5}:%{WORD:text6} %{WORD:text7} %{WORD:text8} %{WORD:decision}"]
remove_field => [ "message" ]
}
date {
match => [ "timestamp", "MM/dd/YYYY HH:mm:ss" ]
remove_field => [ "timestamp" ]
}
}
output {
stdout {
codec => rubydebug
}
}
When the line is received in the log file I am getting a parse error:
{
"path" => "/var/log/tomcat7/catalina.out",
"@timestamp" => 2017-05-24T14:31:18.494Z,
"@version" => "1",
"host" => "acio-web01",
"message" => "ERROR ProcessStatus 05/24/2017 16:31:17 RETC:0 : Request.evaluate:PDP Response decision: Indeterminate",
"type" => "log",
"tags" => [
[0] "_grokparsefailure"
]
}
I am suspicious that the parsing error has something to see with the date format but i have not been able to find a correct way of defining it. Any idea of what I am doing wrong?
Your grok pattern doesn't match your data. In the message it's showing, there are no [] around the date, but your pattern assumes that there are.
You can paste your log line and pattern in at https://grokdebug.herokuapp.com/ and play with it till it matches.