Logstash COMMONAPACHELOG pattern parsing problem

Question

I'm trying to parse the following type of log message:

111.22.333.444 - - [08/Jan/2020:11:50:15 +0100] [https://awdasfe.asfeaf.cas:111] "POST /VFQ3P/asfiheasfhe/v2/safiehjafe/check HTTP/1.1" 204 0 "-" "-" (rt=0.555 urt=0.555 uct=0.122 uht=0.11)

My logstash conf file:

  beats {
    port => 5044
  }
}

filter {
  grok { match => { "message" => "%{IPORHOST:clientip} %{HTTPDUSER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] \[%{NOTSPACE:referrer}\] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-)" } }

  geoip { source => "clientip" }
}


output {
  elasticsearch {
   hosts => ["localhost:9200"]
   index => "my_index5"
  }
}

I'm using almost the same patterns like in the github pattern library for COMMONAPACHELOG. When I put the code through grok debugger in Kibana it works the way I want but when I try to execute it on machine logstash throws me an error that there is a symbol expected before the "(?:%{WORD:verb} part and when I add there \ there is still a problem.

Does anyone have any suggestions for solving the problem?

Thanks in advance!

Answer 1

You have to escape the double quotes (") in your pattern with \, like this:

"%{IPORHOST:clientip} %{HTTPDUSER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] \[%{NOTSPACE:referrer}\] \"(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:response} (?:%{NUMBER:bytes}|-)"

Using the log message you provided, it would result in this:

{
"@version":"1",
"auth":"-",
"host":"******",
"message":"111.22.333.444 - - [08/Jan/2020:11:50:15 +0100] [https://awdasfe.asfeaf.cas:111] \"POST /VFQ3P/asfiheasfhe/v2/safiehjafe/check HTTP/1.1\" 204 0 \"-\" \"-\" (rt=0.555 urt=0.555 uct=0.122 uht=0.11)\r",
"timestamp":"08/Jan/2020:11:50:15 +0100",
"httpversion":"1.1",
"@timestamp":"2020-01-09T13:32:27.442Z",
"verb":"POST",
"response":"204",
"clientip":"111.22.333.444",
"referrer":"https://awdasfe.asfeaf.cas:111",
"ident":"-",
"request":"/VFQ3P/asfiheasfhe/v2/safiehjafe/check",
"bytes":"0"
}