Search code examples
pythonscrapyscrapy-splash

Scrapy: AttributeError: 'str' object has no attribute 'setdefault'

I'm testing out scrapy and splash. I have a spider with:

class MySpider(scrapy.Spider):
    # start_urls = ["http://yahoo.com"]
    name = 'mytest'

    def __init__(self, state='CA', city='San_Francisco', *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.state = state
        self.city = city
        self.start_urls = ['https://trulia.com/{state}/{city}'.format(state=state, city=city)]
        # self.le = LinkExtractor(allow=r'^https://www.trulia.com/property')


    def start_requests(self):
        for url in self.start_urls:
            yield SplashRequest(url, self.parse, endpoint='render.html', args={'wait': 7.5},)

The spider works if I get rid of the init function and uncomment the strt-urls line. stepping through it the problem arises at :

self.start_urls = ['https://trulia.com/{state}/{city}'.format(state=state, city=city)]

traceback:

2019-06-23 09:15:00 [scrapy.core.engine] INFO: Spider opened
2019-06-23 09:15:00 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2019-06-23 09:15:00 [scrapy.utils.signal] ERROR: Error caught on signal handler: <bound method SplashMiddleware.spider_opened of <scrapy_splash.middleware.SplashMiddleware object at 0x
00000292FF8106A0>>
Traceback (most recent call last):
  File "  \lib\site-packages\twisted\internet\defer.py", line 151, in maybeDeferred
    result = f(*args, **kw)
  File "....\lib\site-packages\pydispatch\robustapply.py", line 55, in robustApply
    return receiver(*arguments, **named)
  File "....\lib\site-packages\scrapy_splash\middleware.py", line 245, in spider_opened
    spider.state.setdefault(self.remote_keys_key, {})
AttributeError: 'str' object has no attribute 'setdefault'
2019-06-23 09:15:00 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2019-06-23 09:15:00 [scrapy.core.engine] ERROR: Error while obtaining start requests
Traceback (most recent call last):
  File "....\lib\site-packages\scrapy\core\engine.py", line 127, in _next_request
    request = next(slot.start_requests)
  File "....\lib\site-packages\scrapy_splash\middleware.py", line 165, in process_start_requests
    spider.state.setdefault(self.local_values_key, {})  # fingerprint => value dict

2019-06-23 09:15:00 [scrapy.core.engine] INFO: Spider opened
2019-06-23 09:15:00 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2019-06-23 09:15:00 [scrapy.utils.signal] ERROR: Error caught on signal handler: <bound method SplashMiddleware.spider_opened of <scrapy_splash.middleware.SplashMiddleware object at 0x
00000292FF8106A0>>
Traceback (most recent call last):
  File "  \lib\site-packages\twisted\internet\defer.py", line 151, in maybeDeferred
    result = f(*args, **kw)
  File "....\lib\site-packages\pydispatch\robustapply.py", line 55, in robustApply
    return receiver(*arguments, **named)
  File "....\lib\site-packages\scrapy_splash\middleware.py", line 245, in spider_opened
    spider.state.setdefault(self.remote_keys_key, {})
AttributeError: 'str' object has no attribute 'setdefault'
2019-06-23 09:15:00 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2019-06-23 09:15:00 [scrapy.core.engine] ERROR: Error while obtaining start requests
Traceback (most recent call last):
  File "....\lib\site-packages\scrapy\core\engine.py", line 127, in _next_request
    request = next(slot.start_requests)
  File "....\lib\site-packages\scrapy_splash\middleware.py", line 165, in process_start_requests
    spider.state.setdefault(self.local_values_key, {})  # fingerprint => value dict
AttributeError: 'str' object has no attribute 'setdefault'
2019-06-23 09:15:00 [scrapy.core.engine] INFO: Closing spider (finished)
2019-06-23 09:15:00 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'finish_reason': 'finished',
 'finish_time': datetime.datetime(2019, 6, 23, 13, 15, 0, 838859),
 'log_count/ERROR': 2,
 'log_count/INFO': 9,
 'start_time': datetime.datetime(2019, 6, 23, 13, 15, 0, 822180)}
2019-06-23 09:15:00 [scrapy.core.engine] INFO: Spider closed (finished)

2019-06-23 09:15:00 [scrapy.core.engine] INFO: Closing spider (finished)
2019-06-23 09:15:00 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'finish_reason': 'finished',
 'finish_time': datetime.datetime(2019, 6, 23, 13, 15, 0, 838859),
 'log_count/ERROR': 2,
 'log_count/INFO': 9,
 'start_time': datetime.datetime(2019, 6, 23, 13, 15, 0, 822180)}
2019-06-23 09:15:00 [scrapy.core.engine] INFO: Spider closed (finished)
Solution

  • The problem is your argument name state when you look at the scrapy middleware code

    https://github.com/scrapy-plugins/scrapy-splash/blob/e40ca4f3b367ab463273bee1357d3edfe0601f0d/scrapy_splash/middleware.py#L245

    state has a special meaning for it and it uses dict for the same. So what you need to do is just rename the parameter

    class MySpider(scrapy.Spider):
    # start_urls = ["http://yahoo.com"]
    name = 'mytest'

    def __init__(self, STATE='CA', CITY='San_Francisco', *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.state = state
        self.city = city
        self.start_urls = ['https://trulia.com/{state}/{city}'.format(state=STATE, city=CITY)]
        # self.le = LinkExtractor(allow=r'^https://www.trulia.com/property')


    def start_requests(self):
        for url in self.start_urls:
            yield SplashRequest(url, self.parse, endpoint='render.html', args={'wait': 7.5},)