Search code examples
pythonweb-scrapingbeautifulsoupjupyter

not desired output in web scraping in bs4


I am scraping an product information. But I scrape its price it doesn't give me proper output. There is no error but not the desired output.

And also it produce error while finding the category of a product. Here is my code.

import requests
from bs4 import BeautifulSoup as bs
import pandas

url='https://shop.eziline.com/product/uncategorized/umrah-management-system/'
r=requests.get(url)
soup=bs(r.content,'html.parser')

name=soup.find(class_='product_title entry-title').text.strip()
print(name)
price=soup.find('span',class_='woocommerce-Price-amount amount').text.strip()
print(price)
detail=soup.find(class_='woo-product-details_short-description').text.strip()
print(detail)
category=soup.find('cats-link a').text.strip()
print(category)

Solution

  • You are

    1. picking up the first price rather than the target price. Instead, you can use the main content id as an anchor to the correct section to return the price from

    2. You are trying to use css selector syntax in the last line without applying via the appropriate method and adding in the syntax for a class selector. You could also use category=soup.find(class_='cats-link').a.text.strip()

    Instead:

    import requests
    from bs4 import BeautifulSoup as bs
    
    url='https://shop.eziline.com/product/uncategorized/umrah-management-system/'
    r=requests.get(url)
    soup=bs(r.content,'lxml')
    name=soup.find(class_='product_title entry-title').text.strip()
    print(name)
    price=soup.select_one('#main-content .woocommerce-Price-amount bdi').text.strip()
    print(price)
    detail=soup.find(class_='woo-product-details_short-description').text.strip()
    print(detail)
    category=soup.select_one('.cats-link a').text.strip()
    print(category)