Conversion of text to vector in python

Question

import tokenize
tags = [
  "python, tools",
  "linux, tools, ubuntu",
  "distributed systems, linux, networking, tools",
]
from sklearn.feature_extraction.text import CountVectorizer
vec = CountVectorizer(tokenizer=tokenize)
data = vec.fit_transform(tags).toarray()
print data

I am trying to convert text to vector. But I am facing the following error

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/pymodules/python2.7/sklearn/feature_extraction/text.py", line 398, in fit_transform
term_count_current = Counter(analyze(doc))
File "/usr/lib/pymodules/python2.7/sklearn/feature_extraction/text.py", line 313, in <lambda>
tokenize(preprocess(self.decode(doc))), stop_words)
TypeError: 'module' object is not callable

I have tried to import other libraries too. But nothing seems to be working. How can I correct it?

Answer 1

Not exactly a solution more of a work around from the main page of nltk.org:

>>> import nltk
 >>> sentence = """At eight o'clock on Thursday morning
 ... Arthur didn't feel very good."""
>>> tokens = nltk.word_tokenize(sentence)
>>> tokens
['At', 'eight', "o'clock", 'on', 'Thursday', 'morning',
'Arthur', 'did', "n't", 'feel', 'very', 'good', '.']

Hope this helps