I have this code tested for a sentence and I want to convert it so that I can lemmatize an entire column where each row consists in words without punctuation like: deportivas calcetin hombres deportivas shoes
import wordnet, nltk
nltk.download('wordnet')
from nltk.stem import WordNetLemmatizer
from nltk.corpus import wordnet
import pandas as pd
df = pd.read_excel(r'C:\Test2\test.xlsx')
# Init the Wordnet Lemmatizer
lemmatizer = WordNetLemmatizer()
sentence = 'FINAL_KEYWORDS'
def get_wordnet_pos(word):
"""Map POS tag to first character lemmatize() accepts"""
tag = nltk.pos_tag([word])[0][1][0].upper()
tag_dict = {"J": wordnet.ADJ,
"N": wordnet.NOUN,
"V": wordnet.VERB,
"R": wordnet.ADV}
return tag_dict.get(tag, wordnet.NOUN)
#Lemmatize a Sentence with the appropriate POS tag
sentence = "The striped bats are hanging on their feet for best"
print([lemmatizer.lemmatize(w, get_wordnet_pos(w)) for w in nltk.word_tokenize(sentence)])
Let's suppose Column name is df['keywords'], can you help me use a lambda function in order to lemmatize the entire column like I lemmatize the sentence above?
Many thanks in advance
Here you go:
apply
to apply on the column's sentencessentence
as input and applies the function you wrote, in a similar to how you used in the print statementAs lemmatized keywords:
# Lemmatize a Sentence with the appropriate POS tag
df['keywords'] = df['keywords'].apply(lambda sentence: [lemmatizer.lemmatize(w, get_wordnet_pos(w)) for w in nltk.word_tokenize(sentence)])
As a lemmatized sentence (join
keywords using ' '):
# Lemmatize a Sentence with the appropriate POS tag
df['keywords'] = df['keywords'].apply(lambda sentence: ' '.join([lemmatizer.lemmatize(w, get_wordnet_pos(w)) for w in nltk.word_tokenize(sentence)]))