Logstash GROK parsing of variable length set of key/value pairs

Question

I have a log file that contains a variable number of k/v pairs such as:

2017/01/01 11:11:111,ABC=2,DEF=1
2017/01/01 11:11:112,ABC=4,DEF=1,GHI=7
2017/01/01 11:11:113,ABC=1
2017/01/01 11:11:114,DEF=3,GHI=1...etc

I'm wondering how to handle this so that elasticsearch can use this information. Each key represents a connection pool name and it's value defines how often a pool restart has been initiated since the last time a request to view this information was made.

I'm wanting to report a line graph which would identify over time how pool ABC compares to DEF and GHI...and potentially JKL and MNO ad infinitum.

I'm struggling to find something on this in the documentation.

Answer 1

Could something like this work for you:

filter {
 # filter for pool info
 grok { match => { "message" => "^%{YEAR}\/%{MONTHNUM}\/%{DAY}\s%{HOUR}\:%{MINUTE}\:%{SECOND},%{DATA:poolinfo}"  }
}    
 # parse kv pairs
 ruby { code => '
   fieldArray = event.get( "poolinfo" ).split(",")
   for field in fieldArray
    key = field.split("=")[0]
    value = field.split("=")[1]
    event.set(key, value.to_i)
   end
  '
 }
}