Search code examples
elasticsearchlogstashlogstash-grok

How to parse a log string in Logstash using Grok?

I am trying to parse the following string using Grok;

2018-06-08 13:26:02.002851: <action cmd="run" options="IGNORE_ERROR" path="/usr/lib/vmware/likewise/bin/lw-lsa get-metrics"> (/etc/vmware/vm-support/ad.mfx) took 0.000 sec

I want to separate the above out into columns ultimately like TIMESTAMP, ACTION, OPTIONS, PATH etc - I have tried multiple combinations but have so far failed.

Solution

  • Grok pattern for above log:->

    %{TIMESTAMP_ISO8601:time}:%{SPACE}\<%{WORD:action}%{SPACE} %{DATA:kvpairs}\>%{SPACE}\(%{DATA:path_2}\)%{SPACE}took%{SPACE}%{NUMBER:time_taken}%{SPACE}%{WORD:time_unit}

    In the above grok pattern, I have captured cmd, options and path in an event named kvpairs. This is because these key-value pairs can be easily extracted in logstash using kv filter. So your filter configuration will look like:->

    filter{
    grok(
        match => { "message" => "%{TIMESTAMP_ISO8601:timestamp}:%{SPACE}\<%{WORD:action}%{SPACE} %{DATA:kvpairs}\>%{SPACE}\(%{DATA:path_2}\)%{SPACE}took%{SPACE}%{NUMBER:time_taken}%{SPACE}%{WORD:time_unit}"}
    )

    kv{
        source => "kvpairs"
    }

    date{
        match => ["timestamp","yyyy-MM-dd HH:mm:ss.SSS"]
    }
}

    kv filter by default takes space as the delimiter and will extract columns cmd,options and path. date filter will make the @timestamp variable.