I want to know about the use of recursive function in kv filter. I am using a csv file. I uploaded the file to ES using logstash. After reading the guide from this link https://www.elastic.co/guide/en/logstash/current/plugins-filters-kv.html#plugins-filters-kv-recursive
I came to know that it duplicates the key/values pair and store it in a separate key. But i can't get additional info or examples about the filter. I added a recursive line in logstash config file. No changes. Is it duplicates the fields with values(key-value pairs) or else what this function doing???
Here's my sample csv file data passing through logstash:
"host" => "smackcoders",
"Driveline" => "Four-wheel drive",
"Make" => "Jeep",
"Width" => "79",
"Torque" => "260",
"Year" => "2012",
"Horsepower" => "285",
"City_mpg" => "17",
"Height" => "34",
"Classification" => "Manual,Transmission",
"Model_Year" => "2012 Jeep Wrangler",
"Number_of_Forward_Gears" => "6",
"Length" => "41",
"Highway_mpg" => "21",
"@version" => "1",
"message" => "17,\"Manual,Transmission\",Four-wheel drive,Jeep 3.6L 6 Cylinder 280 hp 260 lb-ft,Gasoline,34,21,285,False,2012 Jeep Wrangler Arctic,41,Jeep,2012 Jeep Wrangler,6,260,6 Speed Manual,79,2012",
"Fuel_Type" => "Gasoline",
"Engine_Type" => "Jeep 3.6L 6 Cylinder 280 hp 260 lb-ft",
"path" => "/home/paulsteven/log_cars/cars.csv",
"Hybrid" => "False",
"ID" => "2012 Jeep Wrangler Arctic",
"@timestamp" => 2019-04-20T07:58:26.552Z,
"Transmission" => "6 Speed Manual"
}
Here's the config file:
input {
file {
path => "/home/paulsteven/log_cars/cars.csv"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
filter {
csv {
separator => ","
columns => ["City_mpg","Classification","Driveline","Engine_Type","Fuel_Type","Height","Highway_mpg","Horsepower","Hybrid","ID","Length","Make","Model_Year","Number_of_Forward_Gears","Torque","Transmission","Width","Year"]
}
kv {
recursive => "true"
}
}
output {
elasticsearch {
hosts => "localhost:9200"
index => "kvfilter1"
document_type => "details"
}
stdout{}
}
Found some examples for recursive in kv filter:
input { generator { count => 1 message => 'foo=1,bar="foor=10,barr=11"' } }
filter {
kv { field_split => "," value_split => "=" recursive => false }
}
will produce
"foo" => "1",
"bar" => "foor=10,barr=11",
whereas
input { generator { count => 1 message => 'foo=1,bar="foor=10,barr=11"' } }
filter {
kv { field_split => "," value_split => "=" recursive => true }
}
will produce
"foo" => "1",
"bar" => {
"foor" => "10",
"barr" => "11"
},