Regex in config for dynamic columns in logstash

I have the log file of which i have pasted two rows below:

Nov 26 14:20:32 172.16.0.1 date=2014-11-26 time=14:18:37 devname=XXXXCCCFFFFF devid=XXXCCVVGFFDD logid=3454363464 type=traffic subtype=forward level=notice vd=root srcip=172.16.1.251 srcport=62032 srcintf="Combo_LAN" dstip=X.X.X.X dstport=X dstintf="wan2" sessionid=16172588 status=close user="X.X" group="Open Group" policyid=2 dstcountry="United States" srccountry="Reserved" trandisp=snat transip=X.X.X.X transport=X service=HTTP proto=6 applist="Block_Applications" duration=11 sentbyte=2377 rcvdbyte=784 sentpkt=6 rcvdpkt=7 identidx=5 utmaction=passthrough utmevent=webfilter utmsubtype=ftgd-cat urlcnt=1 hostname="tacoda.at.atwola.com" catdesc="Advertising"

Nov 26 14:20:32 172.16.0.1 date=2014-11-26 time=14:18:37 devname=XXXXCCCFFFFF devid=XXXCCVVGFFDD logid=3454363464 type=utm subtype=webfilter eventtype=ftgd_allow level=notice vd="root" policyid=2 identidx=5 sessionid=15536743 user="X.X" srcip=X.X.X.X srcport=X srcintf="Combo_LAN" dstip=X.X.X.X dstport=80 dstintf="wan2" service="http" hostname="streaming.sbismart.com" profiletype="Webfilter_Profile" profile="Open Group_Policy" status="passthrough" reqtype="direct" url="/diffusion/" sentbyte=984 rcvdbyte=202 msg="URL belongs to an allowed category in policy" method=domain class=0 cat=18 catdesc="Brokerage and Trading"

My question is i can parse the data if number of columns and order is fixed.

But, how do i parse the dynamic columns in the config file so that i don't get the _grokparsefailure?

Solution

Ruby Plugin can help you.

Here is the configuration:

input {
    stdin{
    }
}

filter {
    ruby {
        code => '
            msg = event["message"]
            msgIndex = msg.index("date=")
            msgInsert = msg[msgIndex..-1]
            msgMap = msgInsert.scan(/(\w+)=("(.*?)"|([^ ]+))/).map { |(first, second)| [first, second] }
            for x in msgMap
                key = x[0]
                value = x[1]
                event[key] = value
            end
        '
    }
}

output {
    stdout{
        codec => rubydebug
    }
}

First, get all the key=value pair by index the start value date=
Then map all the key,value to string array.
Use For loop to insert all the value.

I have try your logs and I can create all the correspond field with the value. Hope this can help you