Search code examples
pythonscrapytimedelaytime-wait

Wait until the webpage loads in Scrapy

I am using scrapy script to load URL using "yield".

MyUrl = "www.example.com"
request = Request(MyUrl, callback=self.mydetail)
yield request
def mydetail(self, response):
    item['Description'] = response.xpath(".//table[@class='list']//text()").extract()
    return item

The URL seems to take minimum 5 seconds to load. So I want Scrapy to wait for some time to load the entire text in item['Description']. I tried "DOWNLOAD_DELAY" in settings.py but no use.

Solution

  • Make a brief view on firebug or another tool to capture responses for Ajax requests, which were made by javascript code. You are able to make a chain of responses to catch those ajax requests which appear after uploading of the page.There are several related questions: parse ajax content, retreive final page, parse dynamic content.