Search code examples
nltkpython-2.6wordnetstemminglemmatization

Stemming some plurals with wordnet lemmatizer doesn't work

Hi i've a problem with nltk (2.0.4): I'm trying to stemming the word 'men' or 'teeth' but it doesn't seem to work. Here's my code:

############################################################################
import nltk
from nltk.corpus import wordnet as wn
from nltk.stem.wordnet import WordNetLemmatizer

lmtzr=WordNetLemmatizer()
words_raw = "men teeth"
words = nltk.word_tokenize(words_raw)
for word in words:
        print 'WordNet Lemmatizer NOUN: ' + lmtzr.lemmatize(word, wn.NOUN)
#############################################################################

This should print 'man' and 'tooth' but instead it prints 'men' and 'teeth'.

any solutions?

Solution

  • I found the solution! I checked the files in wordnet.py the folder /usr/local/lib/python2.6/dist-packages/nltk/corpus/reader and i noticed that the function _morphy(self,form,pos) returns a list containing stemmed words. So i tried to test _morphy :

    import nltk
from nltk.corpus import wordnet as wn
from nltk.stem.wordnet import WordNetLemmatizer

words_raw = "men teeth books"
words = nltk.word_tokenize(words_raw)
for word in words:
        print wn._morphy(word, wn.NOUN)

    This program prints [men,man], [teeth,tooth] and [book]!

    the explanation of why lmtzr.lemmatize () prints only the first element of the list, perhaps it can be found in the function lemmatize, contained in the file 'wordnet.py' which is in the folder /usr/local/lib/python2.6/dist-packages/nltk/stem.

    def lemmatize(self, word, pos=NOUN):
    lemmas = wordnet._morphy(word, pos)
    return min(lemmas, key=len) if lemmas else word

    I assume that it returns only the shorter word contained in the word list, and if the two words are of equal length it returns the first one; for example 'men' or 'teeth'rather than 'man' and 'tooth'