Hi im scraping a webpage with my script, the problem is only one item (title) can be found correctly, other items only throw back html when grabbed like that:
[<Element 'div' class=('rd-product-description__top-accordion-content-description',)>]
My Script:
from requests_html import HTMLSession
url = 'https://www.kaufland.de/product/324406424/?search_value=staubsauger'
s = HTMLSession()
r = s.get(urlos)
r.html.render(sleep=1)
titel = str(r.html.find('h1.rd-title')).replace("[<Element 'h1' title='", "").replace("' class=('rd-title',)>]", "")
price = r.html.find('span.rd-price-information__price')
description = r.html.find('div.rd-product-description__top-accordion-content-description', first=True)
print(description)
The Webpage to scrape: https://www.kaufland.de/product/324406424/?search_value=staubsauger
What is the reason i cant get anything out of it? Trying it with .text also doesnt work, as the element has no text object.
The data you see on the page is stored inside <script> and processed by JavaScript. To get price, description, etc. you can use js2py module. For example:
import re
import js2py
import requests
from bs4 import BeautifulSoup
url = "https://www.kaufland.de/product/324406424/?search_value=staubsauger"
data = re.search(r"__PDPFRONTEND__=(.*?\));", requests.get(url).text).group(1)
data = js2py.eval_js(data)
price = data["state"]["product-store"]["price$"]
desc = data["state"]["product-store"]["description$"]["descriptionHtml"]
desc = BeautifulSoup(desc, "html.parser")
print(price)
print()
print(desc.get_text(strip=True))
Prints:
119.99
Bosch BGN2CHAMP Bodenstaubsauger 600W Aktionsradius 9m Hygienefilter Umschaltbare Rollendüse FilterwechselanzeigeHiSpin MotorStaubsaugen klappt mit einem starken Motor mühelos. Der leistungsstarke und präzise kalibrierte HiSpin Motor sorgt hier für eine gründliche Reinigungsleistung bei geringem Energieverbrauch.Super kompaktStaubsaugen ist jetzt viel einfacher geworden! Dieses kompakte, kleine und leichte Gerät macht
...and so on.