negative lookahead Regexp doesnt work in ES dsl query

Question

The mapping of my Elastic search looks like below:

{
  "settings": {
    "index": {
      "number_of_shards": "5",
      "number_of_replicas": "1"
    }
  },
  "mappings": {
    "node": {
      "properties": {
        "field1": {
          "type": "keyword"
        },
        "field2": {
          "type": "keyword"
        },
        "query": {
          "properties": {
            "regexp": {
              "properties": {
                "field1": {
                  "type": "keyword"
                },
                "field2": {
                  "type": "keyword"
                }
              }
            }
          }
        }
      }
    }
  }
}

Problem is :

I am forming ES queries using elasticsearch_dsl Q(). It works perfectly fine in most of the cases when my query contains any complex regexp. But it totally fails if it contains regexp character '!' in it. It doesn't give any result when the search term contains '!' in it.

For eg:

1.) Q('regexp', field1 = "^[a-z]{3}.b.*") (works perfectly)

2.) Q('regexp', field1 = "^f04.*") (works perfectly)

3.)Q('regexp', field1 = "f00.*") (works perfectly)

4.) Q('regexp', field1 = "f04baz?") (works perfectly)

Fails in below case:

5.) Q('regexp', field1 = "f04((?!z).)*") (Fails with no results at all)

I tried adding "analyzer":"keyword" along with "type":"keyword" as above in the fields, but in that case nothing works.

In the browser i tried to check how analyzer:keyword will work on the input on the case it fails:

http://localhost:9210/search/_analyze?analyzer=keyword&text=f04((?!z).)*

Seems to look fine here with result:

{
  "tokens": [
    {
      "token": "f04((?!z).)*",
      "start_offset": 0,
      "end_offset": 12,
      "type": "word",
      "position": 0
    }
  ]
}

I'm running my queries like below:

search_obj = Search(using = _conn, index = _index, doc_type = _type).query(Q('regexp', field1 = "f04baz?"))
count = search_obj.count()
response = search_obj[0:count].execute()
logger.debug("total nodes(hits):" + " " + str(response.hits.total))

PLease help, its really a annoying problem as all the regex characters work fine in all the queries except !.

Also, how do i check what analyzer is currently applied with above setting in my mappings?

Answer 1

ElasticSearch Lucene regex engine does not support any type of lookarounds. The ES regex documentation is rather ambiguous saying matching everything like .* is very slow as well as using lookaround regular expressions (which is not only ambiguous, but also wrong since lookarounds, when used wisely, may greatly speed up regex matching).

Since you want to match any string that contains f04 and does not contain z, you may actually use

[^z]*fo4[^z]*

Details

• [^z]* - any 0+ chars other than z
• fo4 - fo4 substring
• [^z]* - any 0+ chars other than z.

In case you have a multicharacter string to "exclude" (say, z4 rather than z), you may use your approach using a complement operator:

.*f04.*&~(.*z4.*)

This means almost the same but does not support line breaks:

• .* - any chars other than newline, as many as possible
• f04 - f04
• .* - any chars other than newline, as many as possible
• & - AND
• ~(.*z4.*) - any string other than the one having z4