FreqDist with nltk: ValueError: too many values to unpack

Question

I have been trying to find the frequency distribution of nouns in a given sentence. If I do this:

text = "This ball is blue, small and extraordinary. Like no other ball."
text=text.lower()
token_text= nltk.word_tokenize(text)
tagged_sent = nltk.pos_tag(token_text)
nouns= []
for word,pos in tagged_sent:
    if pos in ['NN',"NNP","NNS"]:
        nouns.append(word)
freq_nouns=nltk.FreqDist(nouns)
print freq_nouns

It considers "ball" and "ball." as separate words. So I went ahead and tokenized the sentence before tokenizing the words:

text = "This ball is blue, small and extraordinary. Like no other ball."
text=text.lower()
sentences = nltk.sent_tokenize(text)                        
words = [nltk.word_tokenize(sent)for sent in sentences]    
tagged_sent = [nltk.pos_tag(sent)for sent in words]
nouns= []
for word,pos in tagged_sent:
    if pos in ['NN',"NNP","NNS"]:
        nouns.append(word)
freq_nouns=nltk.FreqDist(nouns)
print freq_nouns

It gives the following error:

Traceback (most recent call last):
File "C:\beautifulsoup4-4.3.2\Trial.py", line 19, in <module>
for word,pos in tagged_sent:
ValueError: too many values to unpack

What am I doing wrong? Please help.

Answer 1

You were so close!

In this case, you changed your tagged_sent from a list of tuples to a list of lists of tuples because of your list comprehension tagged_sent = [nltk.pos_tag(sent)for sent in words].

Here's some things you can do to discover what type of objects you have:

>>> type(tagged_sent), len(tagged_sent)
(<type 'list'>, 2)

This shows you that you have a list; in this case of 2 sentences. You can further inspect one of those sentences like this:

>>> type(tagged_sent[0]), len(tagged_sent[0])
(<type 'list'>, 9)

You can see that the first sentence is another list, containing 9 items. Well, what does one of those items look like? Well, lets look at the first item of the first list:

>>> tagged_sent[0][0]
('this', 'DT')

If your curious to see the entire object, which I frequently am, you can ask the pprint (pretty-print) module to make it nicer to look at like this:

>>> from pprint import pprint
>>> pprint(tagged_sent)
[[('this', 'DT'),
  ('ball', 'NN'),
  ('is', 'VBZ'),
  ('blue', 'JJ'),
  (',', ','),
  ('small', 'JJ'),
  ('and', 'CC'),
  ('extraordinary', 'JJ'),
  ('.', '.')],
 [('like', 'IN'), ('no', 'DT'), ('other', 'JJ'), ('ball', 'NN'), ('.', '.')]]

So, the long answer is your code needs to iterate over the new second layer of lists, like this:

nouns= []
for sentence in tagged_sent:
    for word,pos in sentence:
        if pos in ['NN',"NNP","NNS"]:
            nouns.append(word)

Of course, this just returns a non-unique list of items, which look like this:

>>> nouns
['ball', 'ball']

You can unique-ify this list in many different ways, but you can quickly by using the set() data structure, like so:

unique_nouns = list(set(nouns))
>>> print unique_nouns
['ball']

For an examination of other ways you can unique-ify a list of items, see the slightly older but extremely useful: http://www.peterbe.com/plog/uniqifiers-benchmark