Handling different log formats in the same file

Question

I have a single log file that contains differing output formats.

For example:

line 1 = 2015-01-1 12:04:56 INFO 192.168.0.1 my_user someone logged in
line 2 = 2015-01-1 12:04:56 WARN [webserver-thread] (MyClass.java:66) user authenticated

Whilst the real solution is to either split them into separate files or unify the formats is it possible to grok differing log formats with Logstash?

Answer 1

My first recommendation is to run one grok{} to strip off the common stuff - the datetime and log level. You can put the remaining stuff back into the [message] field:

%{TIMESTAMP_ISO8601} %{WORD:level} %{GREEDYDATA:message}

Make sure to use the 'overwrite' parameter in grok{}.

Then if you want to parse the remaining information, your (multiple) regexps will be running against a shorter string, which should make them more efficient.

You can then have multiple patterns:

grok {
    match => [
        "message", "PATTERN1",
        "message", "PATTERN2"
    ]
}

By default, grok will stop processing when it hits the first match.