I have a simple Logstash 1.4.1 setup and the data seems correct in stdout, but in ES/Kibana the "message" field is not overwritten.
logstash.conf
input {
file {
path => "/var/log/nginx/access.log"
type => "nginx_access"
}
}
filter {
if [type] == "nginx_access" {
grok {
match => [ "message", "%{NGINXACCESS}" ]
overwrite => [ "message" ]
}
}
}
output {
elasticsearch {
protocol => "http"
}
stdout { codec => rubydebug }
}
NGINX pattern
NGUSERNAME [a-zA-Z\.\@\-\+_%]+
NGUSER %{NGUSERNAME}
NGINXACCESS %{IPORHOST:clientip} %{NGUSER:ident} %{NGUSER:auth} \[%{HTTPDATE:timestamp}\] "%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:response} (?:%{NUMBER:bytes}|-) (?:"(?:%{URI:referrer}|-)"|%{QS:referrer}) %{QS:agent}
stdout
{
"message" => "127.0.0.1 - - [27/Nov/2014:13:43:53 +0100] \"GET /favicon.ico HTTP/1.1\" 502 574 \"-\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.122 Safari/537.36\"",
"@version" => "1",
"@timestamp" => "2014-11-27T12:43:54.230Z",
"type" => "nginx_access",
"host" => "laptop",
"path" => "/var/log/nginx/access.log",
"clientip" => "127.0.0.1",
"ident" => "-",
"auth" => "-",
"timestamp" => "27/Nov/2014:13:43:53 +0100",
"verb" => "GET",
"request" => "/favicon.ico",
"httpversion" => "1.1",
"response" => "502",
"bytes" => "574",
"agent" => "\"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.122 Safari/537.36\""
}
What am I doing wrong? How can I debug?
EDIT
Get API is showing this that fields are in _source, which is not indexed. What to do?
{
"_index":"logstash-2014.11.27",
"_type":"nginx_access4",
"_id":"AUnxR34Z5DzKTB-2TeuG",
"_version":1,
"found":true,
"_source":{
"message":"127.0.0.1 - - [27/Nov/2014:13:43:53 +0100] \"GET / HTTP/1.1\" 502 574 \"-\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.122 Safari/537.36\"",
"@version":"1",
"@timestamp":"2014-11-27T12:43:54.230Z",
"type":"nginx_access",
"host":"laptop",
"path":"/var/log/nginx/access.log",
"clientip":"127.0.0.1",
"ident":"-",
"auth":"-",
"timestamp":"27/Nov/2014:13:43:53 +0100",
"verb":"GET",
"request":"/",
"httpversion":"1.1",
"response":"502",
"bytes":"574",
"agent":"\"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.122 Safari/537.36\""
}
}
Reloading my Index Pattern's field list helped. I created that one before logging any data.