nltk.jaccard_distance function almost always outputs 1.0

Question

I'm trying to get the jaccard distance between two strings of keywords extracted from books. For some reason, the nltk.jaccard_distance() function almost always outputs 1.0

Here is how I preprocess the keywords:

def preprocess(text):
    # make sure to use the right encoding
    text = text.encode("utf-8")
    # remove digits and punctuation
    text = re.sub('[^A-Za-z]+', ' ', text)
    # remove duplicate words
    # note that these aren't sentences, they are strings of keywords
    text = set(text.split())
    text = ' '.join(text)
    # tokenize
    text =  nltk.word_tokenize(text)
    # create sets of n-grams
    text = set(nltk.ngrams(text, n=3))
    return text

Here is where I do the comparison:

def getJaccardSimilarity(keyword_list_1, keyword_list_2):
    keywordstokens_2 = preprocess(keyword_list_2)
    keywordstokens_1 = preprocess(keyword_list_1)
    if len(keywordstokens_1) > 0 and len(keywordstokens_2) > 0:
        return nltk.jaccard_distance(keywordstokens_1, keywordstokens_2)
    else:
        return 0

When I look at the results, the similarity is almost always 1.0, which I thought meant that the n-grams between the two books are identical. Here is some sample data I've just printed out:

KEYWORDS_1: 
set([('laser', 'structur', 'high'), ('high', 'electron', 'halo'), ('atom', 'nuclei', 'helium'), ('nuclei', 'helium', 'neutron'), ('halo', 'atom', 'nuclei'), ('precis', 'laser', 'structur'), ('structur', 'high', 'electron'), ('electron', 'halo', 'atom')])

KEYWORDS_2:
set([('quantum', 'line', 'experi'), ('bench', 'magnet', 'survey'), ('trap', 'tabl', 'quantum'), ('tabl', 'quantum', 'line'), ('use', 'optic', 'trace'), ('line', 'experi', 'cold'), ('trace', 'straight', 'becaus'), ('survey', 'trap', 'tabl'), ('magnet', 'survey', 'trap'), ('straight', 'becaus', 'bench'), ('experi', 'cold', 'requir'), ('optic', 'trace', 'straight'), ('becaus', 'bench', 'magnet')])

SIMILARITY:
1.0

I'm not really sure what I'm missing. Any help is appreciated.

Answer 1

you are calculating the jaccard distance, not the similarity. Hence, it is exactly the other way around: a distance of 0 means your sets are identical, while a distance of 1.0 means their intersection is empty.

Or, to put it differently: similarity(x, y) = 1 - distance(x, y)