Search code examples
pythoncssscrapyselectorchildren

Python: Scrapy Gathering All Text of Selectors Children

I'm trying to scrape the descriptions of ebay listings, and was approaching it with this:

 def parse_description(self, response):
        description = response.css('div#ds_div*::text').get()
        yield {
            "description": description
        }

The idea was to grab the text of all the tags that are under .css('div#ds_div') However I'm getting this as an error:

"Expected selector, got %s" % (peek,))
  File "<string>", line None
cssselect.parser.SelectorSyntaxError: Expected selector, got <DELIM '*' at 10>

Example URL I am trying to scrape: https://www.ebay.co.uk/itm/Vintage-Toastmaster-Chrome-Toaster-Model-D182-4-Slice-Wide-Slot-Nos/114677725765?hash=item1ab3533a45:g:ui8AAOSw-jpgBbFS Where am I going wrong?

Solution

  • The error refers to the selector not being valid:

    div#ds_div*::text

    If you put a space in between the div#ds_div and * it is valid as you've also mentioned in the comments.

    From looking at the link another problem is that the text you're trying to retrieve is inside an iframe with id desc_ifr.

    If you want to scrape the content inside this iframe look at the src attribute of the iframe and scrape this url instead of the url in your question. Then you can do this:

    response.css('div#ds_div p::text').get()