Search code examples
elasticsearchfuzzy-searchfuzzy-comparison

Adjust Fuzziness Level in Percentage in ElasticSearch


I'm working on a project in which I'm using like mine filter, Whenever user will click this he'll get all comments matching with his/her Text(words) My own document is as follows:

{
    "_index": "halohas",
    "_type": "haloha",
    "_id": "5f301d2783cfcb1f016db3e2",
    "_score": 28.848064,
    "_source": {
        "haloha_id": "5f2f33c2c6f0fc27e725ee8a",
        "text": "2-year old sleeping peacefully 😴  ",
        "mood": 2
    }
}

These are results I'm getting:-

{
    "took": 31,
    "timed_out": false,
    "_shards": {
        "total": 5,
        "successful": 5,
        "skipped": 0,
        "failed": 0
    },
    "hits": {
        "total": {
            "value": 36,
            "relation": "eq"
        },
        "max_score": 28.848064,
        "hits": [
            {
                "_index": "halohas",
                "_type": "haloha",
                "_id": "5f301d2783cfcb1f016db3e2",
                "_score": 28.848064,
                "_source": {
                    "haloha_id": "5f2f33c2c6f0fc27e725ee8a",
                    "text": "2-year old sleeping peacefully 😴  ",
                    "mood": 2
                }
            },
            {
                "_index": "halohas",
                "_type": "haloha",
                "_id": "5f2ff43673a23c19f93f3f83",
                "_score": 8.229242,
                "_source": {
                    "haloha_id": "5f2f33c2c6f0fc27e725ee8a",
                    "text": "when wake find 5 year old slipped bed  ",
                    "mood": 2
                }
            },
            {
                "_index": "halohas",
                "_type": "haloha",
                "_id": "5f2ffc6f632e0a21335f34d2",
                "_score": 5.6028643,
                "_source": {
                    "haloha_id": "5f2f33c2c6f0fc27e725ee8a",
                    "text": "view grandmother’s old house. ❤️ ",
                    "mood": 2
                }
            },
            {
                "_index": "halohas",
                "_type": "haloha",
                "_id": "5f30b258a7c85000ec438473",
                "_score": 5.6028643,
                "_source": {
                    "haloha_id": "5f2f33c2c6f0fc27e725ee8a",
                    "text": "streets old Quebec City. 😍 ",
                    "mood": 5
                }
            },
            {
                "_index": "halohas",
                "_type": "haloha",
                "_id": "5f3001608007db1d155f9222",
                "_score": 5.5635624,
                "_source": {
                    "haloha_id": "5f2f33c2c6f0fc27e725ee8a",
                    "text": "dog laying floor sleeping peacefully it’s view little world s bd grateful it. ",
                    "mood": 2
                }
            },
            {
                "_index": "halohas",
                "_type": "haloha",
                "_id": "5f30d0466b4cf850f909f342",
                "_score": 5.0614862,
                "_source": {
                    "haloha_id": "5f2f33c2c6f0fc27e725ee8a",
                    "text": "daughters sleeping beds 😻 ",
                    "mood": 2
                }
            },
            {
                "_index": "halohas",
                "_type": "haloha",
                "_id": "5f30077d93afb8280769fae2",
                "_score": 5.056673,
                "_source": {
                    "haloha_id": "5f2f33c2c6f0fc27e725ee8a",
                    "text": "beach sunny, cold day. ",
                    "mood": 0
                }
            },
            {
                "_index": "halohas",
                "_type": "haloha",
                "_id": "5f30c2d56b7f9d4117391f03",
                "_score": 4.938582,
                "_source": {
                    "haloha_id": "5f2f33c2c6f0fc27e725ee8a",
                    "text": "view children sleeping. ",
                    "mood": 1
                }
            },
            {
                "_index": "halohas",
                "_type": "haloha",
                "_id": "5f30dbf67fc599245178ed02",
                "_score": 4.931981,
                "_source": {
                    "haloha_id": "5f2f33c2c6f0fc27e725ee8a",
                    "text": "sleeping puppies kittens. ",
                    "mood": 2
                }
            },
            {
                "_index": "halohas",
                "_type": "haloha",
                "_id": "5f300fc438bbba64136cd442",
                "_score": 4.6257343,
                "_source": {
                    "haloha_id": "5f2f33c2c6f0fc27e725ee8a",
                    "text": "back deck looking lake.  Never gets old.   ",
                    "mood": 2
                }
            },
            {
                "_index": "halohas",
                "_type": "haloha",
                "_id": "5f3003454d2bcf78c37246e2",
                "_score": 4.43775,
                "_source": {
                    "haloha_id": "5f2f33c2c6f0fc27e725ee8a",
                    "text": "out window, through 2 columns HDB buildings. ",
                    "mood": 2
                }
            },
            {
                "_index": "halohas",
                "_type": "haloha",
                "_id": "5f30547f2242bf327e522f62",
                "_score": 4.2923875,
                "_source": {
                    "haloha_id": "5f2f33c2c6f0fc27e725ee8a",
                    "text": "seeing daughter happy!   ",
                    "mood": 0
                }
            },
            {
                "_index": "halohas",
                "_type": "haloha",
                "_id": "5f3008be88705c67ba1b2be2",
                "_score": 4.133784,
                "_source": {
                    "haloha_id": "5f2f33c2c6f0fc27e725ee8a",
                    "text": "peaceful landscapes mountains, cool, urban cityscape ",
                    "mood": 0
                }
            },
            {
                "_index": "halohas",
                "_type": "haloha",
                "_id": "5f3037d7c7521c22011e5f52",
                "_score": 3.879126,
                "_source": {
                    "haloha_id": "5f2f33c2c6f0fc27e725ee8a",
                    "text": "walking Disney World Park every 2-3 years when go. big thunder lightning rain storm.  ",
                    "mood": 0
                }
            },
            {
                "_index": "halohas",
                "_type": "haloha",
                "_id": "5f2ff9795d4acf75f1130c62",
                "_score": 3.732638,
                "_source": {
                    "haloha_id": "5f2f33c2c6f0fc27e725ee8a",
                    "text": "mountain overlook near home. ",
                    "mood": 1
                }
            },
            {
                "_index": "halohas",
                "_type": "haloha",
                "_id": "5f30b70b385cb141027b0392",
                "_score": 3.6982667,
                "_source": {
                    "haloha_id": "5f2f33c2c6f0fc27e725ee8a",
                    "text": "seeing family together  ",
                    "mood": 2
                }
            },
            {
                "_index": "halohas",
                "_type": "haloha",
                "_id": "5f300b0c5947d73b2f1f2042",
                "_score": 3.4434087,
                "_source": {
                    "haloha_id": "5f2f33c2c6f0fc27e725ee8a",
                    "text": "Something peaceful blue sky ",
                    "mood": 2
                }
            },
            {
                "_index": "halohas",
                "_type": "haloha",
                "_id": "5f3045a0b64c7079e870eb83",
                "_score": 3.2923517,
                "_source": {
                    "haloha_id": "5f2f33c2c6f0fc27e725ee8a",
                    "text": "nature . really enjoy seeing different kinds beauty nature provides  ",
                    "mood": 0
                }
            },
            {
                "_index": "halohas",
                "_type": "haloha",
                "_id": "5f30ad854091a30e3748fdc3",
                "_score": 3.2879872,
                "_source": {
                    "haloha_id": "5f2f33c2c6f0fc27e725ee8a",
                    "text": "seeing family happy  ",
                    "mood": 2
                }
            },
            {
                "_index": "halohas",
                "_type": "haloha",
                "_id": "5f2ffa503f71da190151b6e3",
                "_score": 3.2355409,
                "_source": {
                    "haloha_id": "5f2f33c2c6f0fc27e725ee8a",
                    "text": "standing top Primrose Hill seeing over London😍 ",
                    "mood": 0
                }
            },
            {
                "_index": "halohas",
                "_type": "haloha",
                "_id": "5f309d55495425347e669c72",
                "_score": 3.1025603,
                "_source": {
                    "haloha_id": "5f2f33c2c6f0fc27e725ee8a",
                    "text": "seeing joy faces ones love.  ",
                    "mood": 2
                }
            },
            {
                "_index": "halohas",
                "_type": "haloha",
                "_id": "5f306a8f1e484142ff160182",
                "_score": 2.9667428,
                "_source": {
                    "haloha_id": "5f2f33c2c6f0fc27e725ee8a",
                    "text": "beach, anything calming makes feel peaceful inspired ",
                    "mood": 1
                }
            },
            {
                "_index": "halohas",
                "_type": "haloha",
                "_id": "5f3047beaf93087d444c9522",
                "_score": 2.8672628,
                "_source": {
                    "haloha_id": "5f2f33c2c6f0fc27e725ee8a",
                    "text": "deep night sky, ocean waves, desert canyons... I've got year big tree outside Brooklyn window birdwatching least! ",
                    "mood": 5
                }
            },
            {
                "_index": "halohas",
                "_type": "haloha",
                "_id": "5f2ff9b8d29c5168616b02a2",
                "_score": 2.7855852,
                "_source": {
                    "haloha_id": "5f2f33c2c6f0fc27e725ee8a",
                    "text": "waking seeing fiancé's sleepy face. ♡ ",
                    "mood": 2
                }
            },
            {
                "_index": "halohas",
                "_type": "haloha",
                "_id": "5f302f57c182084e3c52af43",
                "_score": 2.7658296,
                "_source": {
                    "haloha_id": "5f2f33c2c6f0fc27e725ee8a",
                    "text": "very thankful life .i peaceful soul .i come here spreading love happiness.i want anything people. ",
                    "mood": 0
                }
            },
            {
                "_index": "halohas",
                "_type": "haloha",
                "_id": "5f2ff3c74e317d52562f5fe2",
                "_score": 2.3884435,
                "_source": {
                    "haloha_id": "5f2f33c2c6f0fc27e725ee8a",
                    "text": "looking over balcony seeing nice blue ocean  ",
                    "mood": 4
                }
            },
            {
                "_index": "halohas",
                "_type": "haloha",
                "_id": "5f2ff4b38c9d562a32727532",
                "_score": 2.2355409,
                "_source": {
                    "haloha_id": "5f2f33c2c6f0fc27e725ee8a",
                    "text": "seeing smile back me💯🔥🔥🔥 ",
                    "mood": 2
                }
            },
            {
                "_index": "halohas",
                "_type": "haloha",
                "_id": "5f303089c86120678e40dff2",
                "_score": 2.2118597,
                "_source": {
                    "haloha_id": "5f2f33c2c6f0fc27e725ee8a",
                    "text": "looking out over city night seeing all lights cars buildings ",
                    "mood": 2
                }
            },
            {
                "_index": "halohas",
                "_type": "haloha",
                "_id": "5f300cc1ac527721047cc773",
                "_score": 2.2118597,
                "_source": {
                    "haloha_id": "5f2f33c2c6f0fc27e725ee8a",
                    "text": "light rain pouring down mixed thunderstorms seeing view window <3 ",
                    "mood": 6
                }
            },
            {
                "_index": "halohas",
                "_type": "haloha",
                "_id": "5f306a2e55389a57b969edb2",
                "_score": 1.9858739,
                "_source": {
                    "haloha_id": "5f2f33c2c6f0fc27e725ee8a",
                    "text": "nice sunset. usually walk park near house whenever bored, sit grass front beautiful pond. Suchaniceview ",
                    "mood": 2
                }
            },
            {
                "_index": "halohas",
                "_type": "haloha",
                "_id": "5f310e5fe992803f2556f732",
                "_score": 1.8445822,
                "_source": {
                    "haloha_id": "5f2f33c2c6f0fc27e725ee8a",
                    "text": "favourite view view sea when it's sunset. feels calm peaceful eyes see such beautiful scene ",
                    "mood": 2
                }
            },
            {
                "_index": "halohas",
                "_type": "haloha",
                "_id": "5f3013471241a213832e3392",
                "_score": 1.8249211,
                "_source": {
                    "haloha_id": "5f2f33c2c6f0fc27e725ee8a",
                    "text": "view mountains, forest, apple orchard mom's front porch upstate NY. Sometimes there bears, turkey, deer. peaceful! ",
                    "mood": 2
                }
            },
            {
                "_index": "halohas",
                "_type": "haloha",
                "_id": "5f302325a082044a4b271642",
                "_score": 1.7797891,
                "_source": {
                    "haloha_id": "5f2f33c2c6f0fc27e725ee8a",
                    "text": "site seeing nature....*I wrote wanted read other beautiful halohas 😁♥️* ",
                    "mood": 3
                }
            },
            {
                "_index": "halohas",
                "_type": "haloha",
                "_id": "5f307297bec4a071590739b2",
                "_score": 1.6909387,
                "_source": {
                    "haloha_id": "5f2f33c2c6f0fc27e725ee8a",
                    "text": "Vacation:the ocean, now-destroyed deck Hurricane Dorian, Hopetown, Elbow Cay, Bahamas.   Home view woods - serene peaceful. ",
                    "mood": 2
                }
            },
            {
                "_index": "halohas",
                "_type": "haloha",
                "_id": "5f30a16bfd669a03a227f6f3",
                "_score": 1.5437499,
                "_source": {
                    "haloha_id": "5f2f33c2c6f0fc27e725ee8a",
                    "text": "view airplane when really high view seeing all things see everyday different view ✈️✈ ",
                    "mood": 2
                }
            },
            {
                "_index": "halohas",
                "_type": "haloha",
                "_id": "5f30485d9dcac5290736fff2",
                "_score": 1.1237957,
                "_source": {
                    "haloha_id": "5f2f33c2c6f0fc27e725ee8a",
                    "text": "bedroom windows (my bedroom attic) sunset clear night sky. Also view bathroom while shower morning 🌌 ",
                    "mood": 2
                }
            }
        ]
    }
}

When I fetch results with query it gives me all 3 results given above.

My query is as follows:-

{
  "query": {
    "bool": {
      "filter": [
        {
          "match": {
            "haloha_id": "5f2f33c2c6f0fc27e725ee8a"
          }
        }
      ],
      "must": [
        {
          "match": {
            "text": {
              "query": "passing smile each other ",
              "fuzziness": "auto"
            }
          }
        }
      ],
      "should": [
        {
          "term": {
            "mood": {
              "value": 0
            }
          }
        }
      ]
    }
  }
}

In results, if you see at 12th number document is

{
    "_index": "halohas",
    "_type": "haloha",
    "_id": "5f30547f2242bf327e522f62",
    "_score": 4.2923875,
    "_source": {
        "haloha_id": "5f2f33c2c6f0fc27e725ee8a",
        "text": "seeing daughter happy!   ",
        "mood": 0
    }
}

I guess here 3 letters are being matched that is "ing" i.e in my document word is sleeping while in this document word is seeing that is matched on some letter matching based this is what I don't want because both words are too much different.

I want to set some sort of percentage on matching words in fuzziness i.e I want to set fuzziness level. I want no document should return until unless 80% of letters are matched. How can I do that?


Solution

  • Fuzziness is interpreted as a Levenshtein Edit Distance — the number of one character changes that need to be made to one string to make it the same as another string.

    So "sleeping" will match with "seeing" as their is difference of only one character Fuzziness above 2 edit distance is not supported. So value for fuzziness can be only 1 and 2 so it doesn't make sense to allow percentage.

    Check this comment in this issue

    In all cases the behaviour of fuzziness can be reduced to two integers (as fuzziness above 2 isn't supported anyway): the threshold at which it becomes 1, and at which it becomes 2. Percentage fuzziness used to be supported but was removed, presumably because it suggested that there was more flexibility when really it was just being turned into two thresholds.

    Are you suggesting that it should be customisable at the query level, rather than in configuration? If so, some thought would probably need to be put into the syntax, as there could be confusion between settings that define the fuzziness and those that define the length thresholds for fuzziness, which are both integers.