Search code examples
pythonscrapyscrapy-shell

unable to fetch the list values from the website

i fetch all the detail from the desire website but unable to get the some specific information please guide me for that.

targeted domain: https://shop.adidas.ae/en/messi-16-3-indoor-boots/BA9855.html

my code isresponse.xpath('//ul[@class="product-size"]//li/text()').extract()

enter image description here

need to fetch data!!!

enter image description here
Thanks!

Solution

  • Often ecommerce websites have data in json format in page source and then have javscript unpack it on users end.

    In this case you can open up the page source with javascript disabled and search for keywords (like specific size).

    I found in this case it can be found with regular expressions:

    import re
import json
data = re.findall('window.assets.sizesMap = (\{.+?\});', response.body_as_unicode())
json.loads(data[0])
Out: 
{'16': {'uk': '0k', 'us': '0.5'},
 '17': {'uk': '1k', 'us': '1'},
 '18': {'uk': '2k', 'us': '2.5'},
 ...}

    Edit: More accurately you probably want to get different part of the json but nevertheless the answer is more or less the same:

    data = re.findall('window.assets.sizes = (\{(?:.|\n)+?\});', response.body_as_unicode())
json.loads(data[0].replace("'", '"')) # replace single quotes to doubles