Failed to parse query [[local_metadata.host.id]:%{[host.id]}]

Question

I am using Logstash Elasticsearch filter to get the host name using query template as mentioned below -

sample.json

{
  "size": 1,
   "query": {
    "query_string": {
      "query": "[local_metadata.host.id]:%{[host.id]}"
    }
  } }

however, when I test the logstash pipeline, it gives warning mentioned below -

[WARN ] 2023-04-28 09:59:01.047 [[main]>worker1] elasticsearch - Failed to query elasticsearch for previous event {:index=>".fleet-agents", :error=>"[400] {\"error\":{\"root_cause\":[{\"type\":\"query_shard_exception\",\"reason\":\"Failed to parse query [[local_metadata.host.id]:%{[host.id]}]\",\"index_uuid\":\"3_qR6WxhRRugbHtormpQkg\",\"index\":\".fleet-agents-7\"}],\"type\":\"search_phase_execution_exception\",\"reason\":\"all shards failed\",\"phase\":\"query\",\"grouped\":true,\"failed_shards\":[{\"shard\":0,\"index\":\".fleet-agents-7\",\"node\":\"ECbgB995T6OYF-0I-sus6A\",\"reason\":{\"type\":\"query_shard_exception\",\"reason\":\"Failed to parse query [[local_metadata.host.id]:%{[host.id]}]\",\"index_uuid\":\"3_qR6WxhRRugbHtormpQkg\",\"index\":\".fleet-agents-7\",\"caused_by\":{\"type\":\"parse_exception\",\"reason\":\"Cannot parse '[local_metadata.host.id]:%{[host.id]}': Encountered \\\" \\\"]\\\" \\\"] \\\"\\\" at line 1, column 23.\\nWas expecting:\\n    \\\"TO\\\" ...\\n    \",\"caused_by\":{\"type\":\"parse_exception\",\"reason\":\"Encountered \\\" \\\"]\\\" \\\"] \\\"\\\" at line 1, column 23.\\nWas expecting:\\n    \\\"TO\\\" ...\\n    \"}}}}]},\"status\":400}"}

Note: Query in dev tools work fine

GET .fleet-agents/_search
{
 "query": {
   "match": {
     "local_metadata.host.id": "222222-222-2222-22-222"
   }
 } 
}

logstash pipeline

input {
        elasticsearch
        {
        hosts => "localhost"
        user => "reader"
        password => "**************"
        index => "*-test"
        query => '{ "query": {
                             "bool": {
                                      "must": [{"terms": { "kibana.alert.severity": [ "high", "critical"] }} ],
                                      "filter": [ {"range": {"@timestamp": { "gte": "now-2w"}}}]
                                     }
                             }
                  }'
        schedule => "/1 * * * *"
        size => 500
        scroll => "5m"
        docinfo => true
        docinfo_target => "[@metadata][doc]"
        codec => "json"
        }
    }



filter {
elasticsearch {
              hosts => "localhost"
              user => "fleet-user"
              password => "************"
              index => ".fleet-agents"
              query_template => "/data/logstash/pipelines/sample.json"
              fields => { "host.name" => "host_name" }
              }

mutate {
        add_field => {
            "alertHostName" => "%{[host_name]}"
            "alertReason" => "%{kibana.alert.reason}"
            "alertSeverity" => "%{kibana.alert.severity}"
            "alertTime" => "%{kibana.alert.original_time}"

        }
    }

}

output {
stdout {
 codec => "json"
}
}

Could you please assist me with the correct syntax of the match query in query template ?

Thanks in advance!

Answer 1

In Logstash, the nested fields must use the bracket notation not the dotted one

%{[host.id]}

should be

%{[host][id]}

Then your sample.json query is also not syntactically correct, it should be

{
  "size": 1,
   "query": {
    "term": {
      "local_metadata.host.id": "%{[host][id]}"
    }
  } 
}