Script working with IDLE but not PowerShell

Question

I've got this issue where this piece of code:

import nltk

sentence = "Hello everyone on stackoverflow!"
tokens = nltk.word_tokenize(sentence)

print tokens

Throws me this kind of error, when ran from within PowerShell:

    Traceback (most recent call last):
      File "tokenize.py", line 1, in <module>
        import nltk
      File "C:\PYTHON27\lib\site-packages\nltk\__init__.py", line 105, in <module>
        from collocations import *
      File "C:\PYTHON27\lib\site-packages\nltk\collocations.py", line 37, in <module>
        from nltk.util import ingrams
      File "C:\PYTHON27\lib\site-packages\nltk\util.py", line 12, in <module>
        import pydoc
      File "C:\PYTHON27\LIB\pydoc.py", line 55, in <module>
        import sys, imp, os, re, types, inspect, __builtin__, pkgutil, warnings
      File "C:\PYTHON27\LIB\inspect.py", line 39, in <module>
        import tokenize
      File "C:\stuff\tokenize.py", line 4, in <module>
        tokens = nltk.word_tokenize(sentence)
      AttributeError: 'module' object has no attribute 'word_tokenize'

But the same code has no issues in IDLE:

>>> import nltk
>>> sentence = "Hello everyone on stackoverflow!"
>>> tokens = nltk.word_tokenize(sentence)
>>> print tokens
['Hello', 'everyone', 'on', 'stackoverflow', '!']
>>> 

I can't seem to get it fixed no matter what I do. At first, I thought it's the " " space in the directory so I changed it to "stuff", reinstalled nltk but the problem still persisted.

I'd appreciate some help :)

Answer 1

If I correctly read you stack trace, you called your script tokenize.py. But tokenize is allready a module of the Standard Python Library. And still accordingly to the stack trace, the tokenize module is called indirectly from nltk. But as you have a tokenize.py in your current directory Python takes it instead of the standard one.

You should rename your script to a name that does not conflict with the Standard Python Library.