Search code examples
elasticsearchkibanaelastic-stackquerydsl

ElasticSearch Filter not working with must AND must_not


I am trying to filter out errors in Kibana, but not all errors, just errors that do not have a specific string in the message field. My query is as follows:

GET search-apps/_search
{
  "query": {
    "bool": {
      "filter": {
        "bool": {
          "must": [
            {
              "range": {
                "@timestamp": {
                  "gte": "now-120m",
                  "lte": "now"
                }
              }
            },
            {
              "term": {
                "level": "error"
              }
            }
          ],
          "must_not": [
            {
              "term": {
                "message": "Exception thrown while fetching records from Kinesis"
              }
            }
          ]
        }
      }
    }
  }
}

It seems that the filter ignores the "must_not" as I still get errors that have that string in the message field. Is there some sort of order of execution where it is ignoring the must_not? The must works fine, I only get level:error returned, but I also get results that have that string in the message. This also works if I wanted to only return results with the message field:

{
  "query": {
    "match": {
      "message": {
        "query": "Exception thrown while fetching records from Kinesis",
        "type": "phrase"
      }
    }
  }
}

However, per this thread, the opposite doesn't work as suggested.


Solution

  • Your question already contains the solution to your "problem".

    In your must_not-clause you are using a term-query which tests for exact match, whereas in your "test"-query you use a phrase-query. If your message contains the phrase, but also some additional bytes/text, the term-query no longer matches.

    Solution Simply replace term by match_phrase in your bool-query.