Querying keyword field in Logstash

Question

I would like to filter out the string "My Alert" in the message field of my index. The field is a keyword field. To achieve this, I have written the following:

filter {
    mutate {
        add_field => {"secret" => "<my_secret>"}
    }

    elasticsearch {
        hosts => ["<my_host>:9200"]
        ssl => true
        ca_file => "/etc/ssl/certs/ca-certificates.crt"
        index => "<my_index>"
        query => 'message: "My Alert"'
        user => "<my_user>"
        password => "<my_password>" 
        fields => {
            "message" => "[@metadata][message]"
        }
    }

I would then like to put the message field to [@metadata][message], and then process it further via grok patterns. The string I search for is certainly in the message field of some entries of the index, username and password are correct as well. The secret I add via add_field is also correct, as is the hostname, and the ca_file configuration.

I have also tried:

query => 'message: "My Alert*"'
query => 'message: My Alert'
query => 'message: My Alert*'

I did not achieve the desired result with any of those.

Answer 1

First of all the filter elasticsearch is not suited for what you want to achieve:

Elastic docs: Search Elasticsearch for a previous log event and copy some fields from it into the current event. Below are two complete examples of how this filter might be used.

In order to filter based on the message content, you can check these two posts:

• Logstash config, “if message contains…”
• Passing to output index field but getting metadata index

And try something like:

filter {
  if "My Alert" in [message] {
    mutate {
      add_field => { "%[@metadata][message]" => YOUR_MESSAGE_CONTENT }
    }
    grok { YOUR_GROK_HERE }
  }
}