Chaining grok filter patterns for logstash

Question

I am trying to configure logstash to manage my various log sources, one of which is Mongrel2. The format used by Mongrel2 is tnetstring, where a log message will take the form

86:9:localhost,12:192.168.33.1,5:57089#10:1411396297#3:GET,1:/,8:HTTP/1.1,3:200#6:145978#]

I want to write my own grok patterns to extract certain fields from the above format. I received help on this question trying to extract the host. So if in grok-patterns I define

M2HOST ^(?:[^:]*\:){2}(?<hostname>[^,]*)

and then in the logstash conf specify

filter {
  grok {
    match => [ "message", "%{M2HOST}" ]
  }
}

it works as expected. The problem I now have is I want to specify multiple patterns e.g. M2HOST, M2ADDR etc. I tried defining additional ones in the same grok-patterns file

M2HOST ^(?:[^:]*\:){2}(?<hostname>[^,]*)
M2ADDR ^(?:[^:]*\:){3}(?<address>[^,]*)

and changing the logstash conf

filter {
  grok {
    match => [ "message", "%{M2HOST} %{M2ADDR}" ]
  }
}

but now I just get the error _grokparsefailure.

Answer 1

with your sample input from other question and with some guessing about the values names the full match would be:

(?:[^:]*:){2}(?<hostname>[^,]*)[^:]*:(?<address>[^,]*)[^:]*:(?<pid>[^#]*)[^:]*:(?<time>[^#]*)[^:]*:(?<method>[^,]*)[^:]*:(?<query>[^,]*)[^:]*:(?<protocol>[^,]*)[^:]*:(?<code>[^#]*)[^:]*:(?<bytes>[^#]*).*

Producing:

{
  "hostname": [
    [
      "localhost"
    ]
  ],
  "address": [
    [
      "192.168.33.1"
    ]
  ],
  "pid": [
    [
      "57089"
    ]
  ],
  "time": [
    [
      "1411396297"
    ]
  ],
  "method": [
    [
      "GET"
    ]
  ],
  "query": [
    [
      "/"
    ]
  ],
  "protocol": [
    [
      "HTTP/1.1"
    ]
  ],
  "code": [
    [
      "200"
    ]
  ],
  "bytes": [
    [
      "145978"
    ]
  ]
}