I am using logstash 7.6.2. I have log lines that are json strings. Each json has 3 fields, "msg" which is text, "topic" which is text, and "ts" which is a float.
Here is my matching expression:
{"msg"\s*:\s*(?<msg>".*")\s*,\s*"topic"\s*:\s*(?<topic>".*")\s*,\s*"ts"\s*:\s*(?<ts>[+-]?([0-9]*[.])?[0-9]+)\s*}
Here are two example log lines:
{"msg": "2020-05-01 01:09:06,043 ERROR [luna_messaging.handlers.base] HTTP 400: {\"success\": false}\nTraceback (most recent call last):\n File \"/home/lunalife/luna_messaging/handlers/base.py\", line 238, in wrapper\n yield func(self, *args, **kwargs)\n File \"/home/lunalife/.local/lib/python2.7/site-packages/tornado/gen.py\", line 1015, in run\n value = future.result()\n File \"/home/lunalife/.local/lib/python2.7/site-packages/tornado/concurrent.py\", line 237, in result\n raise_exc_info(self._exc_info)\n File \"/home/lunalife/.local/lib/python2.7/site-packages/tornado/gen.py\", line 1021, in run\n yielded = self.gen.throw(*exc_info)\n File \"/home/lunalife/luna_messaging/handlers/device_status.py\", line 41, in get\n raise tornado.web.HTTPError(400, reason=json.dumps(reason))\nHTTPError: HTTP 400: {\"success\": false}", "topic": "com.walker.prod.luna_messaging.handlers.base", "ts": 1588295346.043578}
{"msg": "2020-05-01 01:09:06,076 ERROR [luna_messaging.handlers.base] HTTP 403: Forbidden\nTraceback (most recent call last):\n File \"/home/lunalife/luna_messaging/handlers/base.py\", line 238, in wrapper\n yield func(self, *args, **kwargs)\n File \"/home/lunalife/.local/lib/python2.7/site-packages/tornado/gen.py\", line 1015, in run\n value = future.result()\n File \"/home/lunalife/.local/lib/python2.7/site-packages/tornado/concurrent.py\", line 237, in result\n raise_exc_info(self._exc_info)\n File \"/home/lunalife/.local/lib/python2.7/site-packages/tornado/gen.py\", line 1024, in run\n yielded = self.gen.send(value)\n File \"/home/lunalife/luna_messaging/handlers/device_status.py\", line 46, in get\n raise tornado.web.HTTPError(403)\nHTTPError: HTTP 403: Forbidden", "topic": "com.walker.prod.luna_messaging.handlers.base", "ts": 1588295346.076928}```
I've used a couple of grok testers that show this works. https://grokdebug.herokuapp.com/ and https://grokconstructor.appspot.com/do/match
The problem is, when I integrate into my logstash configuration, it gives me a syntax error. I'm not sure what I am doing wrong.
This is the grok matcher in my logstash configuration:
grok {
match => {"msg"\s*:\s*(?<msg>".*")\s*,\s*"topic"\s*:\s*(?<topic>".*")\s*,\s*"ts"\s*:\s*(?<ts>[+-]?([0-9]*[.])?[0-9]+)\s*}
}
and this is the logstash startup error:
Expected one of [ \\t\\r\\n], \"#\", \"=>\" at line 44, column 21
I believe my matching expression is correct, but I don't know to how add it to the grok config. Any help would be appreciated.
You need to tell the grok filter on which field the pattern matching should be applied.
As you can see from the documentation (https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html#plugins-filters-grok-match), the match-setting follows the syntax
grok{
match => { "FIELDNAME" => "PATTERN" }
}
The default-field Logstash puts the log line text into is called message. So you would adjust your code like so:
grok{
match => { "message" => "PATTERN" }
}
Furthermore, please be aware that the pattern must be quoted and the special characters have to be escaped (I haven't done the latter in the example below). Since you use double quotes in the pattern itself, you need to use single quotes as in the following:
grok{
match => { 'message' => '{"msg"\s*:\s*(?<msg>".*")\s*,\s*"topic"\s*:\s*(?<topic>".*")\s*,\s*"ts"\s*:\s*(?<ts>[+-]?([0-9]*[.])?[0-9]+)\s*}' }
}
I hope I could help you.