Getting logstash log parse error: grokparsefailure

Question

I try to integrate logstash in our application where I include following piece of pattern in custompattern file.

Path: <path>/custom_pattern -- This is custom pattern file. I include this path in conf.
Content: ACCESSLOGPARSE \[%{HTTPDATE:timestamp}\] %{IPORHOST:clientip} (?: xff=%{IPORHOST:xffIp})

My logstash conf file:

input { 
  file{
    path => "/tmp/jboss-logs.log"
    start_position => beginning
  }
}

filter {
  if [path] =~ "jboss" {
    mutate { replace => { "type" => "jboss_access"}}
  grok {
    patterns_dir => "<dir path>"
    match => { "message" => "%{ACCESSLOGPARSE}" }
    }
  }
  date {
    match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
  }
}
output {
  elasticsearch { host => localhost }
  stdout { codec => rubydebug }
}

My jboss-logs.log file content:

[09/Jan/2015:00:00:02 +0000] 127.0.0.1 xff=-
[09/Jan/2015:00:10:17 +0000] 100.20.10.11 xff=100.40.11.3

When I execute logstash, I got following output where the log was not parsed.

{
       "message" => "[09/Jan/2015:00:00:02 +0000] 127.0.0.1 xff=-",
      "@version" => "1",
    "@timestamp" => "2015-01-20T15:30:10.865Z",
          "host" => "Salvador",
          "path" => "/tmp/jboss-logs.log",
          "type" => "jboss_access",
          "tags" => [
        [0] "_grokparsefailure"
    ]
}
{
       "message" => "[09/Jan/2015:00:10:17 +0000] 100.20.10.11 xff=100.40.11.3",
      "@version" => "1",
    "@timestamp" => "2015-01-20T15:30:10.869Z",
          "host" => "Salvador",
          "path" => "/tmp/jboss-logs.log",
          "type" => "jboss_access",
          "tags" => [
        [0] "_grokparsefailure"
    ]
}

The problem is the 'xff' key in log may contain an ip or '-'. I have tried with following patterns too. But they too did't work.

ACCESSLOGPARSE \[%{HTTPDATE:timestamp}\] %{IPORHOST:clientip} (?: xff=%{IPORHOST:xffIp}|-) 
and
ACCESSLOGPARSE \[%{HTTPDATE:timestamp}\] %{IPORHOST:clientip} (?: xff=%{IPORHOST:xffIp}|xff=-)

What is wrong with the parser for this pattern?

Answer 1

Your first pattern only uses IPORHOST, which doesn't support "-" as a valid value.

Your second pattern (?: xff=%{IPORHOST:xffIp}|-) is looking for "xff=1.2.3.4" or "-". Your input is "xff=-", which doesn't match.

Also note that the space after "(?:" is significant and should be removed.

These work:

xff=(?:%{IPORHOST:xffIp}|-)

(but the xffIp will be NULL when the value is "-")

Use a more generic pattern:

(?:xff=%{NOTSPACE:xffIp})

or you could define a new pattern:

IPORHOSTORDASH (?:%{IPORHOST}|-)

and use it:

(?:xff=%{IPORHOSTORDASH:xffIp})

to put the parsed value into the xffIP field.

If you have more key/value fields on your line, you should look into the kv{} filter.