Search code examples
scrapyscrapy-splash

Using scrapy shell with splash return empty value

I'm scraping this site: http://www.germandeli.com/Meats/Sausages which contains some dynamic content.

I am using scrapy shell with splash to render the javascript but it returns the empty value []. My system is Ubuntu 14.04 LTS.

Here the code I used:

$ scrapy shell 'http://localhost:8050/render.html?url=http://www.germandeli.com/Meats/Sausages'
>>> response.xpath('*//h2[@class="item-cell-name"]/a/@href').extract()

Any hint would be greatly appreciated!

Solution

  • I figured it out. I forgot to add '&timeout=10&wait=5' in the end of the link!

    scrapy shell 'http://localhost:8050/render.html?url=http://www.germandeli.com/Meats/Sausages&timeout=10&wait=5'