Search code examples
pythonpandasdataframewordnetsynset

Get the synonyms out of a dataframe

I have a dataset that is comprised of {question, answer} for a chatbot training, I loaded it with pandas. I'm trying to get a bag of synonyms for each word in each question with wordnet.synsets. and I'am having some difficulities doing so, here is the attempt that i've tried. 

import pandas  as pd`
import nltk.corpus
from nltk.corpus import stopwords, wordnet
from nltk.tokenize import word_tokenize
from nltk.stem import PorterStemmer, WordNetLemmatizer
df =pd.read_csv('healthtapQAs++.csv')
df['question']=df['question'].str.pad(width= i,side= 'left')
df['unpunctuated'] = df['question'].str.replace(r'[^\w\s]+', '')
df['tokenized'] = df['unpunctuated'].apply(word_tokenize) 
df['synonyms'] = df['tokenized'].apply(lambda x: [wordnet.synsets(y) for y 
in x])
df['synonyms_beta'] = df['synonyms'].apply( lambda x:[(y[0].name()) for y in 
x])`

and this is the type of error that i keep getting 

>   df['synonyms_beta'] = df['synonyms'].apply( lambda x:[(y[0].name()) for y in x])

IndexError: list index out of range
Solution

  • You may try:

    df['synonyms_beta'] = df['synonyms'].apply( lambda x:[(y[0].name()) if len(y) >0 else "no_syn" for y in x])