I have a KMeans clustering script and it organises some documents based on the contents of the text. The documents fall into 1 of 3 clusters, but it seems very YES or NO, I'd like to be able to see how releveant to the cluster each document is.
eg. Document A is in Cluster 1 90% matching, Document B is in Cluster 1 but 45% matching.
Therefore I can create some sort of threshold to say, I only want documents 80% or higher.
dict_of_docs = {'Document A':'some text content',...'Document Z':'some more text content'}
# Vectorizing the data, my data is held in a Dict, so I just want the values.
vectorizer = TfidfVectorizer(stop_words='english')
X = vectorizer.fit_transform(dict_of_docs.values())
X = X.toarray()
# 3 Clusters as I know that there are 3, otherwise use Elbow method
# Then add the vectorized data to the Vocabulary
NUMBER_OF_CLUSTERS = 3
km = KMeans(
n_clusters=NUMBER_OF_CLUSTERS,
init='k-means++',
max_iter=500)
km.fit(X)
# First: for every document we get its corresponding cluster
clusters = km.predict(X)
# We train the PCA on the dense version of the tf-idf.
pca = PCA(n_components=2)
two_dim = pca.fit_transform(X)
scatter_x = two_dim[:, 0] # first principle component
scatter_y = two_dim[:, 1] # second principle component
plt.style.use('ggplot')
fig, ax = plt.subplots()
fig.set_size_inches(20,10)
# color map for NUMBER_OF_CLUSTERS we have
cmap = {0: 'green', 1: 'blue', 2: 'red'}
# group by clusters and scatter plot every cluster
# with a colour and a label
for group in np.unique(clusters):
ix = np.where(clusters == group)
ax.scatter(scatter_x[ix], scatter_y[ix], c=cmap[group], label=group)
ax.legend()
plt.xlabel("PCA 0")
plt.ylabel("PCA 1")
plt.show()
order_centroids = km.cluster_centers_.argsort()[:, ::-1]
# Print out top terms for each cluster
terms = vectorizer.get_feature_names()
for i in range(3):
print("Cluster %d:" % i, end='')
for ind in order_centroids[i, :10]:
print(' %s' % terms[ind], end='')
print()
for doc in dict_of_docs:
text = dict_of_docs[doc]
Y = vectorizer.transform([text])
prediction = km.predict(Y)
print(prediction, doc)
I don't believe it is possible to do exactly what you want because k-means is not really a probabilistic model and its scikit-learn implementation (which is what I'm assuming you're using) just doesn't provide the right interface.
One option I'd suggest is to use the KMeans.score method, which does not provide a probabilistic output but provides a score that is larger the closer a point is to the closest cluster. You could threshold by this, such as by saying "Document A is in cluster 1 with a score of -.01 so I keep it" or "Document B is in cluster 2 with a score of -1000 so I ignore it".
Another option is to used the GaussianMixture model instead. A gaussian mixture is a very similar model to k-means and it provides the probabilities you want with GaussianMixture.predict_proba.