Search code examples
scrapyweb-crawlerscrapy-splashscrapy-shell

Enable to select element using Scrapy shell

I'm trying to print out all the titles of the products of this website using scrapy shell: 'https://www.woolworths.com.au/shop/browse/drinks/cordials-juices-iced-teas/iced-teas'

Once it is open I start fetching:

fetch('https://www.woolworths.com.au/shop/browse/drinks/cordials-juices-iced-teas/iced-teas')

And I try to print out the title of each product as a result nothing is selected:

>>> response.css('.shelfProductTile-descriptionLink::text')
output:  []

Also tried: 

>>> response.css('a')
output:  []

How can I do ? Thanks

Solution

  • Your code is correct. What happens is that there is no a element in the HTML retrieved by scrapy. When you visit the page with your browser, the product list is populated with javascript, on the browser side. They are not in the HTML code.

    In the doc you'll find techniques to pre-render javascript. Maybe you should try that.