Search code examples
pythonpandasmatrixscipy

Python Pandas Distance matrix using jaccard similarity

I have implemented a function to construct a distance matrix using the jaccard similarity:

import pandas as pd
entries = [
    {'id':'1', 'category1':'100', 'category2': '0', 'category3':'100'},
    {'id':'2', 'category1':'100', 'category2': '0', 'category3':'100'},
    {'id':'3', 'category1':'0', 'category2': '100', 'category3':'100'},
    {'id':'4', 'category1':'100', 'category2': '100', 'category3':'100'},
    {'id':'5', 'category1':'100', 'category2': '0', 'category3':'100'}
           ]
df = pd.DataFrame(entries)

and the distance matrix with scipy

from scipy.spatial.distance import squareform
from scipy.spatial.distance import pdist, jaccard

res = pdist(df[['category1','category2','category3']], 'jaccard')
squareform(res)
distance = pd.DataFrame(squareform(res), index=df.index, columns= df.index)

The problem is that my result looks like this which seems to be false:

enter image description here

What am i missing? The similarity of 0 and 1 have to be maximum for example and the other values seem wrong too

Solution

  • Looking at the docs, the implementation of jaccard in scipy.spatial.distance is jaccard dissimilarity, not similarity. This is the usual way in which distance is computed when using jaccard as a metric. The reason for this is because in order to be a metric, the distance between the identical points must be zero.

    In your code, the dissimilarity between 0 and 1 should be minimized, which it is. The other values look correct in the context of dissimilarity as well.

    If you want similarity instead of dissimilarity, just subtract the dissimilarity from 1.

    res = 1 - pdist(df[['category1','category2','category3']], 'jaccard')