Custom GROK filter - Logstash -> Elasticsearch

Question

I have a log which is getting captured and sent to logstash, the format of the log is

22304999    5   400.OUTPUT_SERVICE.510  submit  The limit has been exceeded. Please use a different option. 2.54.44.221 /api/output/v3/contract/:PCID/order /api/output/v3/contract/:pcid/order https://www.example.org/output/ PUT 400 2017-09-28T15:50:57.843176Z

I am trying to create a custom grok filter to add the header fields before it gets sent to elasticsearch.

My aim is something like this,

 SessionID   => "22304999"
 HitNumber   => "5"
 FactValue   => "400.OUTPUT_SERVICE.510"
 DimValue1   => "submit"
 ErrMessage  => "The limit has been exceeded. Please use a different option."
 IP          => "2.54.44.221"
 TLT_URL     => "/api/output/v3/contract/:PCID/order"
 URL         => "/api/output/v3/contract/:pcid/order"
 Refferer    => "https://www.example.org/output/"
 Method      => "PUT"
 StatsCode   => "400"
 ReqTime     => "2017-09-28T15:50:57.843176Z"

I am new to this so only trying to understand how I apply and test this, for example I would start with an empty filter,

filter {
   grok {
     match => { "message" => "" }
   }
 }

My first question is match => { "message" => "" } is message just a log line? What defines 'message'?

My log and the fields I want are separated by a Tab, after each Tab its a new field, would this make what I am trying to achieve easier, rather than looking for a pattern can I just look for the next Tab?

Failing this, could someone provide an example for one of my fields, from that I should be able to complete the rest.

Answer 1

Regex: (?<SessionID>\S+)\s+(?<HitNumber>\S+)\s+(?<FactValue>\S+)\s+(?<DimValue1>\S+)\s+(?<ErrMessage>.+)\s+(?<IP>(?:\d{1,3}\.){3}\d{1,3})\s+(?<TLT_URL>\S+)\s+(?<URL>\S+)\s+(?<Refferer>\S+)\s+(?<Method>\S+)\s+(?<StatsCode>\S+)\s+(?<ReqTime>\S+)

Details:

• (?<>) Named Capture Group
• \S matches any non-whitespace character
• \d Matches a digit, {n,m} Matches between n and m times
• + Matches between one and unlimited times

Regex demo

Output:

{
  "SessionID": [
    [
      "22304999"
    ]
  ],
  "HitNumber": [
    [
      "5"
    ]
  ],
  "FactValue": [
    [
      "400.OUTPUT_SERVICE.510"
    ]
  ],
  "DimValue1": [
    [
      "submit"
    ]
  ],
  "ErrMessage": [
    [
      "The limit has been exceeded. Please use a different option."
    ]
  ],
  "IP": [
    [
      "2.54.44.221"
    ]
  ],
  "TLT_URL": [
    [
      "/api/output/v3/contract/:PCID/order"
    ]
  ],
  "URL": [
    [
      "/api/output/v3/contract/:pcid/order"
    ]
  ],
  "Refferer": [
    [
      "https://www.example.org/output/"
    ]
  ],
  "Method": [
    [
      "PUT"
    ]
  ],
  "StatsCode": [
    [
      "400"
    ]
  ],
  "ReqTime": [
    [
      "2017-09-28T15:50:57.843176Z"
    ]
  ]
}