I have setup the environment variables as instructed by the official NLTK wiki. I run into the following error in my very first example. Here are the codes:
from nltk.tokenize import StanfordSegmenter
datapath = "D:/Coding/stanford-segmenter/"
corporadict = datapath+"data/"
modelpath = datapath + "data/pku.gz"
dictpath = datapath + "data/dict-chris6.ser.gz"
segmenter = StanfordSegmenter(path_to_sihan_corpora_dict=corporadict,path_to_model=modelpath,path_to_dict=dictpath)
res = segmenter.segment(u"这是斯坦福中文分词器")
but Python gives me the following error. Traceback (most recent call last):
File "D:/Video data/data_processed/ugctext/test_stanford.py", line 19, in <module>
res = segmenter.segment(u"这是斯坦福中文分词器")
File "C:\Python35\lib\site-packages\nltk\tokenize\stanford_segmenter.py", line 164, in segment
return self.segment_sents([tokens])
File "C:\Python35\lib\site-packages\nltk\tokenize\stanford_segmenter.py", line 192, in segment_sents
stdout = self._execute(cmd)
File "C:\Python35\lib\site-packages\nltk\tokenize\stanford_segmenter.py", line 211, in _execute
stdout, _stderr = java(cmd, classpath=self._stanford_jar, stdout=PIPE, stderr=PIPE)
File "C:\Python35\lib\site-packages\nltk\internals.py", line 129, in java
p = subprocess.Popen(cmd, stdin=stdin, stdout=stdout, stderr=stderr)
File "C:\Python35\lib\subprocess.py", line 947, in __init__
restore_signals, start_new_session)
File "C:\Python35\lib\subprocess.py", line 1198, in _execute_child
args = list2cmdline(args)
File "C:\Python35\lib\subprocess.py", line 751, in list2cmdline
needquote = (" " in arg) or ("\t" in arg) or not arg
TypeError: argument of type 'NoneType' is not iterable
can anyone help me with this issue? Thanks!
For some reason list2cmdline(args) in subprocess.py is returning [None], and it is not being handled properly. I would guess that it's a problem with the java() call in stanford_segmenter.py.
From here you can see that the code was updated to require Java 8 back in 2014. If your Java version is lower than this, it could be the problem.