Python - Issue with the dimension of array in cdist function

Question

I am trying to find the right number of cluster for k-means and using the cdist function for this.

I can understand the argument for cdist should be of same dimension. I tried printing the size of both argument which is (2542, 39) and (1, 39).

Can someone please suggest where i am going wrong ?

print(tfidf_matrix.shape) ### Returning --> (2542, 39)
#Finding optimal no. of clusters
from scipy.spatial.distance import cdist
clusters=range(1,10)
meanDistortions=[]

for k in clusters:
    model=KMeans(n_clusters=k)
    model.fit(tfidf_matrix)
    prediction=model.predict(tfidf_matrix)
    print(model.cluster_centers_.shape)  ## Returning (1, 39)
    meanDistortions.append(sum(np.min(cdist(tfidf_matrix, model.cluster_centers_, 'euclidean'), axis=1)) /
                           tfidf_matrix.shape[0])

Error:

ValueError                                Traceback (most recent call last)
<ipython-input-181-c15e32d863d2> in <module>()
     12     prediction=model.predict(tfidf_matrix)
     13     print(model.cluster_centers_.shape)
---> 14     meanDistortions.append(sum(np.min(cdist(tfidf_matrix, model.cluster_centers_, 'euclidean'), axis=1)) /
     15                            tfidf_matrix.shape[0])
     16 

~\Downloads\Conda\envs\data-science\lib\site-packages\scipy\spatial\distance.py in cdist(XA, XB, metric, *args, **kwargs)
   2588 
   2589     if len(s) != 2:
-> 2590         raise ValueError('XA must be a 2-dimensional array.')
   2591     if len(sB) != 2:
   2592         raise ValueError('XB must be a 2-dimensional array.')

ValueError: XA must be a 2-dimensional array.

Answer 1

It probably is a type issue.

Tfidf probably is not a dense matrix as required by cdist. Of course it makes sense to use a sparse matrix here.

However, cdist does not seem to accept sparse matrixes: scipy cdist with sparse matrices