I am trying to create my own tagged corpus on demonetization dataset and the dataset has approximately 6250 tweets. the code is below, although it is giving the results for a small dataset of size 200 entries.
df = pd.read_csv('Demonetization_data29th2.csv',encoding = "ISO-8859-1")
text = df['CONTENT']
sentiment = df['sentiment']
a =[]
tagged = [[nltk.word_tokenize(sent)] for sent in df['CONTENT']]
tagged = [nltk.pos_tag(sent) for sent in tagged]
print tagged[0]
print "---------"
brown_tagged_sents = tagged
print brown_tagged_sents[0]
size = int(len(brown_tagged_sents) * 0.7)
tags = [tag for (word, tag) in brown.tagged_words()]
defaultTag = nltk.FreqDist(tags).max()
print defaultTag
train_sents = brown_tagged_sents[:size]
test_sents = brown_tagged_sents[size:]
tagger = ngramTagger(train_sents, 2, defaultTag)
print tagger.evaluate(test_sents)
i got an error like :
File "C:/Users/HP/pos-2.py", line 41, in <module>
tagged = [[nltk.word_tokenize(sent)] for sent in df['CONTENT']]
File "C:\ProgramData\Anaconda2\lib\site-packages\nltk\tokenize\__init__.py", line 130, in word_tokenize
sentences = [text] if preserve_line else sent_tokenize(text, language)
File "C:\ProgramData\Anaconda2\lib\site-packages\nltk\tokenize\__init__.py", line 97, in sent_tokenize
return tokenizer.tokenize(text)
File "C:\ProgramData\Anaconda2\lib\site-packages\nltk\tokenize\punkt.py", line 1235, in tokenize
return list(self.sentences_from_text(text, realign_boundaries))
File "C:\ProgramData\Anaconda2\lib\site-packages\nltk\tokenize\punkt.py", line 1283, in sentences_from_text
return [text[s:e] for s, e in self.span_tokenize(text, realign_boundaries)]
File "C:\ProgramData\Anaconda2\lib\site-packages\nltk\tokenize\punkt.py", line 1274, in span_tokenize
return [(sl.start, sl.stop) for sl in slices]
File "C:\ProgramData\Anaconda2\lib\site-packages\nltk\tokenize\punkt.py", line 1314, in _realign_boundaries
for sl1, sl2 in _pair_iter(slices):
File "C:\ProgramData\Anaconda2\lib\site-packages\nltk\tokenize\punkt.py", line 312, in _pair_iter
prev = next(it)
File "C:\ProgramData\Anaconda2\lib\site-packages\nltk\tokenize\punkt.py", line 1287, in _slices_from_text
for match in self._lang_vars.period_context_re().finditer(text):
TypeError: expected string or buffer
Please guide me where I am going wrong
you probably got a null in one or more rows of that column.
fill it with empty string like:
df['CONTENT'] = df['CONTENT'].fillna('')