Search code examples
pythondnsscrapyscreen-scraping

how to scrapy handle dns lookup failed

I am looking to handle a DNS error when scraping domains Scrapy.

Here's the error that I am seeing:

ERROR: Error downloading <GET http://domain.com>: DNS lookup failed: address 'domain.com' not found [Errno 8] nodename nor servname provided, or not known.

How could I be notified when I get an error like this, so that I can handle it myself without Scrapy just throwing an error and moving on.

Solution

  • Use errback along with callback:

    Request(url, callback=your_callback, errback=your_errorback)

    and errback:

    def your_errorback(self, response):
    //your logic will be here