Search code examples
logstash

logstash add_field is not populating value instead it is hardcoding syntax in index


I am trying to create new output index using 3 input index. In new output index I need to populate few specific fields from input index. I am trying to create new field using add_field.it is hardcoding like '%{[index1name][field1inIndex1]}' instead populating value from index. I have tried below code:

input
{
elasticsearch{
hosts => ["hostname"]
index => "index1"
query => '{"query":{"match_all":{}}}'
docinfo => "true"
user => "uname"
password =>"pwd"
ssl=>"true"
}
elasticsearch {
#same like above for index2
}
elasticsearch {
#same like above for index3
}
}    
filter
{
mutate
{
add_field =>["newfieldname","%{[index1][fieldinindex1]}"]
}
}
output 
{
elasticsearch {
#creating new index here
}
}

Solution

  • If a sprintf reference is not substituted then it indicates the field does not exist on the event.

    The index name is not added to the field name by the elasticsearch input. (It may be added as part of [@metadata] if you enable the docinfo option.) So unless the name of the field on the document in the index your are reading from contains the index name what you need is

    mutate { add_field => { "newfieldname" => "%{[fieldinindex1]}" } }
    

    If you do want the index name in [newfieldname] then you have to use a reference to it, for example

    mutate { add_field => { "newfieldname" => "%{[@metadata][_index]}_%{[fieldinindex1]}" } }