Hi I want to create spider that scraps one website a day. I have a spider that scraps all the stuff I need but I need to implement a pause after each article scraped. I have tried threading
module and the time
module as well but using them doesn't seem to work as I get this response (from only some of the requests):
DEBUG: Retrying <GET https://www.example.com/.../> (failed 1 times): [<twisted.python.failure.Failure twisted.internet.error.ConnectionLost: Connection to the other side was lost in a non-clean fashion: Connection lost.>]
.
class AutomatedSpider(scrapy.Spider):
name = 'automated'
allowed_domains = ['example-domain.com']
start_urls = [
'https://example.com/page/1/...'
]
pause = threading.Event()
article_num = 1
def parse(self, response):
for page_num in range(1, 26):
for href in set(response.css(".h-100 a::attr(href)").extract()):
# extract data from all the articles on current page
self.pause.wait(5.0) # this causes the response mentioned above
yield scrapy.Request(href, callback=self.parse_article)
self.article_num += 1
# move to next page
next_page = 'https://www.information-age.com/page/'+str(page_num)+'/...'
yield scrapy.Request(next_page, callback=self.parse)
def parse_article(self, response):
# function to extract desired data from website that is being scraped
I don't think time.sleep and waits from threading can work well in Scrapy due to its asynchronous way of working. What you can do is the following: