tf-idf for text cluster-analysis

Question

I would like to group small texts included in a column, df['Texts'], from a dataframe. An example of sentences to analyse are as follows:

    Texts

  1 Donald Trump, Donald Trump news, Trump bleach, Trump injected bleach, bleach coronavirus.
  2 Thank you Janey.......laughing so much at this........you have saved my sanity in these mad times. Only bleach Trump is using is on his heed 🤣
  3 His more uncharitable critics said Trump had suggested that Americans drink bleach. Trump responded that he was being sarcastic.
  4 Outcry after Trump suggests injecting disinfectant as treatment.
  5 Trump Suggested 'Injecting' Disinfectant to Cure Coronavirus?
  6 The study also showed that bleach and isopropyl alcohol killed the virus in saliva or respiratory fluids in a matter of minutes.

Since I know that TF-IDF is useful for clustering, I have been using the following lines of code (by following some previous questions in the community):

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.cluster import KMeans
import re
import string

def preprocessing(line):
    line = line.lower()
    line = re.sub(r"[{}]".format(string.punctuation), " ", line)
    return line

tfidf_vectorizer = TfidfVectorizer(preprocessor=preprocessing)
tfidf = tfidf_vectorizer.fit_transform(all_text)

kmeans = KMeans(n_clusters=2).fit(tfidf) # the number of clusters could be manually changed

However, since I am considering a column from a dataframe, I do not know how to apply the above function. Could you help me with it?

Answer 1

def preprocessing(line):
    line = line.lower()
    line = re.sub(r"[{}]".format(string.punctuation), " ", line)
    return line

tfidf_vectorizer = TfidfVectorizer(preprocessor=preprocessing)
tfidf = tfidf_vectorizer.fit_transform(df['Texts'])

kmeans = KMeans(n_clusters=2).fit(tfidf)

You just need to replace all_text with your df. It'd be nice to build a pipeline first then apply vectorizer and Kmeans all at the same time.

Also for getting more precise result more preprocessing of your text is never a bad idea. In addition, however I dont think lowering the text is a good idea since you naturally remove a good feature for style of writing(If we consider you want to find the author or assign author to a group) but for getting the sentiment of sentences yeah it's better to lower.