pythonweb-scrapingbeautifulsouppython-requestsscrapinghub
How to scrape the heading and description using python and beautiful soup?
Overview of the problem:
Link : https://www.bobfinancial.com/eterna.jsp
In the Details Section: Basically I want all Points.
details:
[ #This is an array of Strings...
"Milestone Rewards: Earn 10,000 bonus reward points on spending ₹ 50,000 within 60 days & 20,000 bonus reward points on spending ₹ 5,00,000 in a year.",
"Fuel Surcharge Waiver*: 1% fuel surcharge waiver at all fuel stations across India on transactions between Rs.400 and Rs.5,000 (Max. Rs. 250 per statement cycle). Note -No Reward Points are earned on fuel transactions.",
"Core Reward Points: 3 reward points for every ₹ 100 spent on any other category.",
"Redeem reward points for cashback: Redeem your reward points as cashback and other exciting options.All your accumulated reward points can be redeemed for cashback @ 1 reward point = ₹ 0.25",
"In-built insurance cover: Get free Personal Accidental Death Cover to ensure financial protection of your family (Air: 1 Crs, Non-Air: 10 Lakhs) ",
"Zero liability on lost card. Report loss of card immediately to ensure zero liability on any fraudulent transactions",
"Easy EMI option. Convert purchase of > 2,500/- on your card into easy EMIs of 6/12 months"
]
Actual Problem :
Link to scrape : https://www.online.citibank.co.in/credit-card/rewards/citi-rewards-credit-card?eOfferCode=INCCCCTWAFCTRELM
Items that I want from that link are there in the following images :
from urllib.request import urlopen
from bs4 import BeautifulSoup
import json,requests
details = []
url = ['https://www.online.citibank.co.in/credit-card/rewards/citi-rewards-credit-card?eOfferCode=INCCCCTWAFCTRELM']
html = urlopen(url[0])
soup = BeautifulSoup(html, 'lxml')
addr= soup.find_all('span',class_ = 'm-bottom-0 header-4 font-weight-bold display-text')
print(addr)
I wrote the above code and got stuck after I got this output :
[]
I don't know how to proceed further and scrape the information that I want, any help is highly appreciated.
Solution
You could search for just the headings which have a display-text class. The body is then just the following <span> entry. This avoids having any hard coded offsets which could break for other similar pages. For example:
from bs4 import BeautifulSoup
import requests
url = ['https://www.online.citibank.co.in/credit-card/rewards/citi-rewards-credit-card?eOfferCode=INCCCCTWAFCTRELM']
html = requests.get(url[0])
soup = BeautifulSoup(html.content, 'lxml')
data = []
for span in soup.select('span.m-bottom-0.display-text.font-weight-bold'):
data.append([span.get_text(strip=True), span.find_next('span').get_text(strip=True)])
print(data)
This would give you a data structure holding the headers and the descriptions as follows:
[
['Citi Rewards Credit Card', 'Make your shopping more rewarding'],
['Make your shopping more rewarding', 'Get up to 2500 welcome reward points*'],
['Accelerated rewards', 'Earn minimum 1 reward point for every 125 on all purchases.\nEarn 10X reward points at online and physical department and apparel stores.'],
['Bonus rewards', 'Get 300 bonus points on card purchase of INR 30,000 or more in a month'],
['Evergreen reward points', "Redeem now or keep collecting – it's a choice you have because your reward points will never expire."],
['Tap n Pay', 'Now pay the easy way by enabling contactless payments on your Citi credit card.Click hereto see how.'],
['Rewards', "Redeem rewards: Redeem now or keep collecting – it's a choice you have.Click hereto see how"],
['Travel and Lifestyle Services', 'Contact the Travel and Lifestyle specialist to create and plan an experience that will help you enjoy the best in life. Simply call 0008 004 407 027 for Mastercard®cardholders or 1800-114-999 for Visa cardholders.']
]
You can skip over the first two results (if not needed) with print(data[2:])
It can be extended to multiple URLs and the header and description can be combined:
from bs4 import BeautifulSoup
import requests
urls = [
'https://www.online.citibank.co.in/credit-card/fuel-card/citi-indianoil-card?eOfferCode=INCCCCTWAFCTIOPLM',
'https://www.online.citibank.co.in/credit-card/rewards/citi-rewards-credit-card?eOfferCode=INCCCCTWAFCTRELM',
]
for url in urls:
html = requests.get(url)
soup = BeautifulSoup(html.content, 'lxml')
data = []
for span in soup.select('span.m-bottom-0.display-text.font-weight-bold'):
data.append(f'{span.get_text(strip=True)}: {span.find_next("span").get_text(strip=True)}')
# Display the list of strings
print('\n'.join(data[1:]))
print()
Giving you the following output for the two URLs:
Accelerated Rewards on Fuel spends: 4 Turbo points / Rs. 150 spent &1% fuel surcharge reversalon fuel purchase atauthorized IndianOil outlets^
Earn on all Daily Spends: 2 Turbo points / Rs. 150 spent on groceries and supermarkets#1 Turbo point / Rs. 150 on all other spends.#Clickherefor the details
Redeem Instantly: 1 Turbo Point = Re. 1 of free fuelRedeem your Turbo points instantly via SMS for freeFuel atauthorized IndianOil outlets^
Tap n Pay: Now pay the easy way by enabling contactless payments on your Citi credit card.Click hereto see how.
Redeem Rewards!: Choose from a range of redeeming options including fuel, holidays, air miles, cash back and more.Click hereto see how
Dining Privileges: Up to 20% savings across participating restaurants. To find a Citibank partner restaurant near you,click here
Travel and Lifestyle Services: Contact the Travel and Lifestyle specialist to create and plan an experience that will help you enjoy the best in life. Simply call 0008 004 407 027 for Mastercard cardholders or 1800-114-999 for Visa cardholders.
Accelerated rewards: Earn minimum 1 reward point for every 125 on all purchases.
Earn 10X reward points at online and physical department and apparel stores.
Bonus rewards: Get 300 bonus points on card purchase of INR 30,000 or more in a month
Evergreen reward points: Redeem now or keep collecting – it's a choice you have because your reward points will never expire.
Tap n Pay: Now pay the easy way by enabling contactless payments on your Citi credit card.Click hereto see how.
Rewards: Redeem rewards: Redeem now or keep collecting – it's a choice you have.Click hereto see how
Travel and Lifestyle Services: Contact the Travel and Lifestyle specialist to create and plan an experience that will help you enjoy the best in life. Simply call 0008 004 407 027 for Mastercard®cardholders or 1800-114-999 for Visa cardholders.