Search code examples
stringrecordsimilaritylevenshtein-distancelinkage

Record linkage using String similarity Techniques

We are working on Record linkage project. We are observing a strange behavior from all of the standard technique like Jaro Winkler, Levenshtein, N-Gram, Damerau-Levenshtein, Jaccard index, Sorensen-Dice

Say, String 1= MINI GRINDER KIT
String 2= Weiler 13001 Mini Grinder Accessory Kit, For Use With Small Right Angle Grinders
String 3= Milwaukee Video Borescope, Rotating Inspection Scope, Series: M-SPECTOR 360, 2.7 in 640 x 480 pixels High-Resolution LCD, Plastic, Black/Red

In the above case string 1 and string 2 are related the score of all the methods as shown below.
Jaro Winkler -> 0.391666651
Levenshtein -> 75
N-Gram, -> 0.9375
Damerau -> 75
Jaccard index -> 0
Sorensen-Dice -> 0
Cosine -> 0

But string 1 and string 3 are not at all related, but distance method are giving very high score.
Jaro Winkler -> 0.435714275
Levenshtein -> 133
N-Gram, -> 0.953571439
Damerau -> 133
Jaccard index -> 1
Sorensen-Dice -> 0
Cosine -> 0

Any thoughts .?

Solution

  • All distance calculation score are case sensitive. Hence bring all of them to same case. Then you get to see the score calculation appropriately.