Search code examples
regexlogstashlogstash-grok

How to match a pattern of "a=b c=d" with changing order in grok (logstash)?

I'm using Logstash to match Fortinet analyzer logs, and the problem is there are so many pattern without order of the fields.

e.g. one type of message would be:

service=DNS hostname="a.b.net" profile="Dns" action=blocked reqtype=direct url="/" sentbyte=0 rcvdbyte=0 direction=N/A msg="URL belongs to a denied category in policy" method=domain cat=61 catdesc="Phishing" crscore=60 crlevel=high

...and another is:

msg="File is infected." action=blocked service=HTTP sessionid=33137 direction=incoming filename="favicon.ico" quarskip=No-skip virus="MSWord/Agent.DD60!tr" dtype="Virus" ref="http://www.fortinet.com/ve?vn=MSWord%2FAgent.DD60%21tr" virusid=6920465 profile="AV"

As you can see both have msg, action, service and profile but with different order.

Is there anyway to build a pattern to match something like: 

(.*?)=%{DATA:\1?}\s

...while giving the field the name of the first match?

Solution

  • Use the kv{} filter which can split it all apart and doesn't care about the order.