Search code examples
pythonscrapysplash-screenscrapy-splash

Scrapy Splash is returning an empty list


I am learning Scrapy Splash right now and build a very basic spider via a tutorial.

But running the spider scrapy crawl quotes -o results_file.csv it is only returning an empty csv file.

It also says it crawled 0 sites, so it seems not to find the webpage?

I am using Windows 10 home

I double checked the xpath expressions and splash is working on http://localhost:8050/

I know I am missing something really simple here... can you help?

I put this into the settings.py

    DOWNLOADER_MIDDLEWARES = {
    'scrapy_splash.SplashCookiesMiddleware': 723,
    'scrapy_splash.SplashMiddleware': 725,
    'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware': 810,
    }
SPLASH_URL = 'http://localhost:8050/'   

DUPEFILTER_CLASS = 'scrapy_splash.SplashAwareDupeFilter'
HTTPCACHE_STORAGE   = 'scrapy_splash.SplashAwareFSCacheStorage'

And this is the Spider I am running:

import scrapy
from scrapy_splash import SplashRequest

class QuotesSpider(scrapy.Spider):
    name = 'quotes'
    allowed_domains = ['quotes.toscrape.com']
    start_urls = ['http://quotes.toscrape.com/js/']

    def start_requests(self):
        for url in self.start_urls:
            yield SplashRequest(url=url,
                                callback=self.parse,
                                endpoint='render.html')


    def parse(self, response):
        quotes = response.xpath('//*[@class="quote"]')
        for quote in quotes:
            yield {'author': quote.xpath('.//*[@class="author"]/text()').extract_first(),
                   'quote': quote.xpath('.//*[@class="text"]/text()').extract_first()}

This is the Output I get:

2022-01-02 15:32:45 [scrapy.utils.log] INFO: Scrapy 2.4.1 started (bot: quotes_spider)
2022-01-02 15:32:45 [scrapy.utils.log] INFO: Versions: lxml 4.7.1.0, libxml2 2.9.12, cssselect 1.1.0, parsel 1.5.2, w3lib 1.21.0, Twisted 21.7.0, Python 3.9.7 (default, Sep 16 2021, 16:59:28) [MSC v.1916 64 bit (AMD64)], pyOpenSSL 21.0.0 (OpenSSL 1.1.1f  31 Mar 2020), cryptography 36.0.0, Platform Windows-10-10.0.19042-SP0
2022-01-02 15:32:45 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.selectreactor.SelectReactor
2022-01-02 15:32:45 [scrapy.crawler] INFO: Overridden settings:
{'BOT_NAME': 'quotes_spider',
 'DUPEFILTER_CLASS': 'scrapy_splash.SplashAwareDupeFilter',
 'HTTPCACHE_STORAGE': 'scrapy_splash.SplashAwareFSCacheStorage',
 'NEWSPIDER_MODULE': 'quotes_spider.spiders',
 'SPIDER_MODULES': ['quotes_spider.spiders']}
2022-01-02 15:32:45 [scrapy.extensions.telnet] INFO: Telnet Password: 4cc7c27ecac6bf1d
2022-01-02 15:32:45 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.feedexport.FeedExporter',
 'scrapy.extensions.logstats.LogStats']
2022-01-02 15:32:45 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy_splash.SplashCookiesMiddleware',
 'scrapy_splash.SplashMiddleware',
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2022-01-02 15:32:45 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2022-01-02 15:32:45 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2022-01-02 15:32:45 [scrapy.core.engine] INFO: Spider opened
2022-01-02 15:32:45 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2022-01-02 15:32:45 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2022-01-02 15:32:46 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://quotes.toscrape.com/> (referer: None)
2022-01-02 15:32:46 [scrapy.core.engine] INFO: Closing spider (finished)
2022-01-02 15:32:46 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 223,
 'downloader/request_count': 1,
 'downloader/request_method_count/GET': 1,
 'downloader/response_bytes': 2200,
 'downloader/response_count': 1,
 'downloader/response_status_count/200': 1,
 'elapsed_time_seconds': 0.569004,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2022, 1, 2, 14, 32, 46, 398295),
 'log_count/DEBUG': 1,
 'log_count/INFO': 10,
 'response_received_count': 1,
 'scheduler/dequeued': 1,
 'scheduler/dequeued/memory': 1,
 'scheduler/enqueued': 1,
 'scheduler/enqueued/memory': 1,
 'start_time': datetime.datetime(2022, 1, 2, 14, 32, 45, 829291)}
2022-01-02 15:32:46 [scrapy.core.engine] INFO: Spider closed (finished)

Edit

So it is not the code itself - thanks for clearing that!

Docker + Splash is running @ 0.0.0.0:8050

Is there anything I need to configure for Splash+Scrapy in Windows Home I might have missed for it to run properly?


Solution

  • It's currently working for me so the issue may be with your settings, see if mine help:

    
    BOT_NAME = 'test_again' #change to yours
    
    SPIDER_MODULES = ['test_again.spiders'] #change to yours
    NEWSPIDER_MODULE = 'test_again.spiders' #change to yours
    SPLASH_URL = 'http://localhost:8050'
    
    # Crawl responsibly by identifying yourself (and your website) on the user-agent
    USER_AGENT = 'my-cool-project (http://example.com)'
    
    # Obey robots.txt rules
    ROBOTSTXT_OBEY = False
    
    
    DEFAULT_REQUEST_HEADERS = {
    'User-Agent':'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36'
    }
    
    SPIDER_MIDDLEWARES = {
    'scrapy_splash.SplashDeduplicateArgsMiddleware': 100,
    }
    
    DOWNLOADER_MIDDLEWARES = {
    'scrapy_splash.SplashCookiesMiddleware': 723,
    'scrapy_splash.SplashMiddleware': 725,
    'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware': 810,
    }
    DUPEFILTER_CLASS = 'scrapy_splash.SplashAwareDupeFilter'
    
    

    Output:

    {'author': 'Albert Einstein', 'quote': '“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”'}
    2022-01-02 18:43:06 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/>
    {'author': 'J.K. Rowling', 'quote': '“It is our choices, Harry, that show what we truly are, far more than our abilities.”'}
    2022-01-02 18:43:06 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/>
    {'author': 'Albert Einstein', 'quote': '“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”'}
    2022-01-02 18:43:06 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/>
    
    ...