I have most recent logstash installed (2.3), and I want to parse syslog using grok filter. So, here is the filter:
filter {
if [type] == "linux-syslog" {
grok {
match => { "message" => "^%{SYSLOGTIMESTAMP:syslog_timestamp}\s*%{SYSLOGHOST:syslog_hostname}\s*(%{PROG:syslog_program})?\s*(:?\[%{POSINT:syslog_pid}\])?:?\s*%{GREEDYDATA:syslog_message}[a-z]*\s*$" }
patterns_dir => ["/var/opt/logstash/patterns"]
add_tag => "syslog_everything"
keep_empty_captures => "true"
}
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}
This is what I get from logstash trying to parse it:
"message" => "Apr 20 14:31:35 node1 ansible-service: Invoked with name=logstash pattern=None enabled=True state=restarted sleep=None arguments= runlevel=default ",
"@version" => "1",
"@timestamp" => "2016-04-20T14:31:35.000Z",
"path" => "/var/log/syslog",
"host" => "node1",
"type" => "linux-syslog",
"syslog_timestamp" => "Apr 20 14:31:35",
"syslog_hostname" => "node1",
"syslog_program" => "ansible-service:",
"syslog_pid" => nil,
"syslog_message" => "Invoked with name=logstash pattern=None enabled=True state=restarted sleep=None arguments= runlevel=default ",
"tags" => [
[0] "syslog_everything",
[1] "_grokparsefailure"
]
}
And...
{ "message" => "Apr 20 14:35:10 node1 crontab[29052]: (vagrant) END EDIT (vagrant)", "@version" => "1", "@timestamp" => "2016-04-20T14:35:10.000Z", "path" => "/var/log/syslog", "host" => "node1", "type" => "linux-syslog", "syslog_timestamp" => "Apr 20 14:35:10", "syslog_hostname" => "node1", "syslog_program" => "crontab", "syslog_pid" => "29052", "syslog_message" => "(vagrant) END EDIT (vagrant)", "tags" => [ [0] "syslog_everything", [1] "_grokparsefailure" ] }
What am I doing wrong here? I checked this using grokdebugger and it passed fine...
If the fields are being created, then the grok is working. You probably have another grok stanza that is failing. Add a distinct tag_on_failure to each one.