Search code examples
pythonmodulescrapypython-importscrapy-splash

Scrapy splash^ AttributeError: 'module' object has no attribute 'Spider'


I don't know why, but there's been an error recently:

File "C:\Users\name\PycharmProjects\splash\project\project\spiders\scrapy.py", line 5, in <module>
    class ScrapySpider(scrapy.Spider):
AttributeError: 'module' object has no attribute 'Spider'

My full code:

import scrapy
from scrapy_splash import SplashRequest

class Spider(scrapy.Spider):
        name = "spide"
        start_urls = [
        many links
        ]
def start_requests(self):
        for url in self.start_urls:
        yield SplashRequest(url, self.parse,
              endpoint='render.html',
              args={'wait': 0.5},
        )

def parse(self, response):
        "my parsed info"

What's the problem and how can it be solved?

P.S. I set up ScrapySplash through this tutorial


Solution

  • The problem is in the name of your python file:

    C:\Users\name\PycharmProjects\splash\project\project\spiders\scrapy.py
    

    When you run

    import scrapy
    

    Python is trying to import the current file (that you've written your code in) as the scrapy module. To solve this, rename your file to something other than scrapy.py. This will mean that when Python tries to import scrapy, it will import the scrapy.py file in your site-packages folder, assuming you've installed scrapy correctly (pip install scrapy).