Search code examples
regexlogstash-groklogstash-configurationgrok

Grok Pattern to Parse Multiple loglines

I have 3 lines of logs with different structure. I am trying to construct a grok pattern to filter the logs.

[2022-10-04 21:45:27,444: INFO/MainProcess] Events of group {task} enabled by remote
[2022-10-04 21:43:06,521: ERROR/MainProcess] consumer: Cannot connect to redis://10.0.13.57:6379/0: Error 111 connecting to 10.0.13.34:6379. Connection refused..
[2022-10-04 21:45:22 +0000] [3094] [INFO] Listening at: http://0.0.0.0:8793 (3094)

I am expecting:

timestamp: loglevel: message:

The grok pattern I have doesn't match anything: \[%{TIMESTAMP_ISO8601:timestamp}\]\:%{LOGLEVEL:loglevel}%{WORD: class} %{SPACE}%{GREEDYDATA:logMessage}

Solution

  • You need to have two grok pattern for separate logs.

    [2022-10-04 21:45:27,444: INFO/MainProcess] Events of group {task} enabled by remote
[2022-10-04 21:43:06,521: ERROR/MainProcess] consumer: Cannot connect to redis://10.0.13.57:6379/0: Error 111 connecting to 10.0.13.34:6379. Connection refused..

    The grok pattern for the above two logs:

    %{DATESTAMP:timestamp}\: %{LOGLEVEL:loglevel}\/%{DATA:data}\] %{GREEDYDATA:message}

    Output: enter image description here

    [2022-10-04 21:45:22 +0000] [3094] [INFO] Listening at: http://0.0.0.0:8793 (3094)

    The grok pattern for the above log:

    \[%{TIMESTAMP_ISO8601:timestamp} \+%{DATA:data}\] \[%{LOGLEVEL:loglevel}\] %{GREEDYDATA:message}

    Output: enter image description here

    Also, you can make use of the Drop Filter of the logstash to drop the field data generated [see the output screenshot below] after parsing your logs using the above GROK pattern.