Import error urllib.parse in scrapy

Question

I am trying to use scrapy. I did pip install scrapy. My python version is 2.7.9. After installing when I typed scrapy in the terminal it gave the following error:

  File "/usr/bin/scrapy", line 7, in <module>
    from scrapy.cmdline import execute
  File "/usr/lib/python2.7/site-packages/scrapy/__init__.py", line 48, in <module>
    from scrapy.spiders import Spider
  File "/usr/lib/python2.7/site-packages/scrapy/spiders/__init__.py", line 10, in <module>
    from scrapy.http import Request
  File "/usr/lib/python2.7/site-packages/scrapy/http/__init__.py", line 10, in <module>
    from scrapy.http.request import Request
  File "/usr/lib/python2.7/site-packages/scrapy/http/request/__init__.py", line 12, in <module>
    from scrapy.utils.url import escape_ajax
  File "/usr/lib/python2.7/site-packages/scrapy/utils/url.py", line 9, in <module>
    from six.moves.urllib.parse import (ParseResult, urlunparse, urldefrag,
ImportError: No module named urllib.parse

I know scrapy needs python 2.7 but urllib.parse is introduced in python 3, before that it was urlparse. Looking at the error it seems the error is in the scrapy installation. What to do? I have uninstalled and reinstalled scrappy several times. But the problem is still there.

Answer 1

We can use seperate imports for python 2 and python 3.

try:
    from urllib.parse import urlparse
except ImportError:
    from urlparse import urlparse

Got this answer from here. no module named urllib.parse (How should I install it?)