pythonselenium-webdriverweb-scrapingbeautifulsoupinfinite-scroll
How to use python selenium to locate the scrollbar on the right and then scroll down continuously to fully load all items content?
I want to scrape all the items' information from this page https://allinone.pospal.cn/m#/categories, refer to the screenshot.
the content is loading dynamically when I scroll down the right side of the page to the bottom and this webpage has two scrollbars. I tried a few times; however, I still failed to scroll it, so I can't get the item information on the next pages (next scroll), only able to extract the first 20 items, but it has about 1500+ items.
please kindly help me with this.
my code is as below:
import time
import requests
from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
# Load the webpage
url = 'https://allinone.pospal.cn/m#/categories'
driver = webdriver.Chrome()
driver.get(url)
# Wait for the promotion image to load and click it
promotion_image = WebDriverWait(driver, 10).until(
EC.presence_of_element_located(
(By.XPATH, '//img[@src="//imgw.pospal.cn/we/westroe/img/categories/discount.png"]'))
)
promotion_image.click()
# get focus on the right side of the page (two scroll bar, focus on the right)
# Wait for the div to load and get focus on the element
items_div = WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.CSS_SELECTOR, 'div.yb-scrollable'))
)
#items_div.send_keys(Keys.NULL)
items_div.click()
# Do something with the div, e.g. get its text content
#@print(items_div.text)
#-
#### attempt 1 - to scroll
##Scroll to the bottom of the page
# scroll_pause_time = 1
# scroll_step = 500
# scroll_height = driver.execute_script("return Math.max( document.body.scrollHeight, document.body.offsetHeight, document.documentElement.clientHeight, document.documentElement.scrollHeight, document.documentElement.offsetHeight );")
# while True:
# driver.execute_script(f"window.scrollTo(0, {scroll_height});")
# scroll_height_new = driver.execute_script("return Math.max( document.body.scrollHeight, document.body.offsetHeight, document.documentElement.clientHeight, document.documentElement.scrollHeight, document.documentElement.offsetHeight );")
# if scroll_height_new == scroll_height:
# break
# scroll_height = scroll_height_new
# time.sleep(scroll_pause_time)
# -
# -
#### attempt 2
# """A method for scrolling to the bottom of the page."""
# # Get scroll height.
# last_height = driver.execute_script("return document.body.scrollHeight")
# while True:
# # Scroll down to the bottom.
# driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
# # Wait to load the page.
# time.sleep(2)
# # Calculate new scroll height and compare with last scroll height.
# new_height = driver.execute_script("return document.body.scrollHeight")
# if new_height == last_height:
# break
# last_height = new_height
#### attempt 3
lenOfPage = driver.execute_script("window.scrollTo(0, document.body.scrollHeight);var lenOfPage=document.body.scrollHeight;return lenOfPage;")
match=False
while(match==False):
lastCount = lenOfPage
time.sleep(3)
lenOfPage = driver.execute_script("window.scrollTo(0, document.body.scrollHeight);var lenOfPage=document.body.scrollHeight;return lenOfPage;")
if lastCount==lenOfPage:
match=True
# Extract the content of the yb-item tags
soup = BeautifulSoup(driver.page_source, 'html.parser')
yb_items = soup.find_all('div', {'class': 'yb-item'})
for yb_item in yb_items:
print(yb_item.text.strip())
# Close the browser window
driver.quit()
Solution
You don't need selenium to scrape this web page.
You can use this POST API endpoint, if you go to network Tab
https://allinone.pospal.cn/wxapi/product/ListMulti
This will return response in a json format and you can use you required fields.
I have used 10 pages only also page side increased to 50, you can increase based on your requirements.
import requests
from bs4 import BeautifulSoup as soup
header={'User-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36'}
url="https://allinone.pospal.cn/wxapi/product/ListMulti"
for page in range(10):
data={
"cUids": 10,
"pageIdx": page,
"size": 50
}
response=requests.post(url,data=data,headers=header,cookies ={'cookie' :'uuvid=BFcFXAwyBDUDZAA4XmNab1FiV2FdPlswUWdTYl01BWFXYVQzXG8ANgNjUT8NYg08BTFQMFsyXWIAYFEzXWMHZgRmBTM='}).json()
print("data for page {}".format(page+1))
for item in response['data']:
print("name : " + item['name'])
print("Sell price :{}".format(item['sellPrice']))
print("===========================")
Output:
data for page 1
name : 2015 sena(375ml) * 6
Sell price :3240.0
name : 義大利嘉雅 布雷西卡(長相思)白酒 2017 Gaja Alteni di Brassica *6
Sell price :5700.0
name : 法國捷高正牌(美人魚)2009 CHA. Giscours Margaux
Sell price :4500.0
name : 凱歌皇牌香槟(冰袋) VEUVE CLICQUOT Brut Ice Jacket *6
Sell price :2520.0
name : 金貝克 瓊瑤漿白酒 *6
Sell price :1200.0
name : 法國沙龍帝皇珍藏香檳 *6
Sell price :2297.0
name : 法國凱歌玫瑰香檳 Veuve *6
Sell price :2975.0
name : Bollinger 伯蘭爵香檳 *6
Sell price :2662.0
name : 布根地塞繆爾比洛夏布利一級園白酒2018 Samuel Billaud *6
Sell price :2923.0
name : Cloudy bay雲霧黑皮諾 *6
Sell price :1931.0
name : Cloudy bay雲霧霞多麗 *6
Sell price :1775.0
name : 布根地樂弗萊夫 夏山-蒙哈榭園白酒 2019 Olivier Leflaive *6
Sell price :4855.0
name : 布根地樂弗萊夫 普里尼-蒙哈榭一級園白酒 2019 Olivier Leflaive *6
Sell price :6160.0
name : 義大利嘉雅 思波斯園紅酒2010 Gaja Barolo Sperss *6
Sell price :15480.0
name : 法國薩賓女神香檳 *6
Sell price :2349.0
name : 法國玫瑰正牌2008年Chateau Montrose Saint-Estephe*6
Sell price :6360.0
name : 法國玫瑰正牌2012年Chateau Montrose Saint-Estephe *6
Sell price :5340.0
name : 5cl藍帶 *6
Sell price :540.0
name : 5cl拱橋 *6
Sell price :930.0
name : 5cl名仕 *6
Sell price :300.0
name : 2017 智利阿帕塔莊園Clos Apalta*12
Sell price :11640.0
name : 法國夢伯爵白酒2016年 *6
Sell price :3240.0
name : 法國太保正牌2014年Chateau Talbot *6
Sell price :3180.0
name : 法國白馬副牌2011年 *6
Sell price :10800.0
name : 美國作品一號副牌(序曲) OPUS One Overture N.V. ＊6
Sell price :7980.0
name : 碧崗巴朗酒莊正牌 2014 Ch. Pichon Baron *6
Sell price :7440.0
name : Le Petit Lion Du Marouis De Las Cases法國雄獅副牌2009*6
Sell price :3420.0
name : 意大利羅密歐布魯耐羅精選紅酒2012年 Castello Romitorio *6
Sell price :8340.0
name : 安東尼 薩拉榭瓦羅白酒2020 Antinori Cervaro Della Sala *6
Sell price :4560.0
name : Chateau Branaire-Ducru法國周伯通正牌2012 *6
Sell price :3120.0
name : 克拉吉 黑皮諾Craggy Range *6
Sell price :2346.0
name : 克拉吉頂級黑皮諾Craggy Range *6
Sell price :4802.0
name : 意大利 皮特羅酒莊 加拉托納頂級梅洛紅酒2012 Petrolo Galatrona Valdarn
Sell price :5460.0
name : Chateau Calon Segur 法國卡龍沙高(情人的心)1996*6
Sell price :7500.0
name : 意大利聖菲利波路酒莊 珍藏布魯奈羅 *6
Sell price :9480.0
name : sena2014 *6
Sell price :5700.0
name : sena2018 *6
Sell price :5940.0
name : sena2016 *6
Sell price :5880.0
name : 375ml 法國利是金副牌2015 Chevalier de Lascombes *24
Sell price :3840.0
name : 法國酩悅香檳(單支) MOET Chandon Brut Imperial NV *6
Sell price :1920.0
name : 智利阿帕塔apalta2016 *6
Sell price :5220.0
name : 2017 智利阿帕塔莊園Clos Apalta *6
Sell price :6060.0
name : 美國黑駿馬 *12
Sell price :4920.0
name : Gaja嘉雅 卡瑪康達普拉米(藍標) *6
Sell price :2220.0
name : 美國山脊霞多麗白酒 *6
Sell price :3360.0
name : 南非 利斯摩維歐尼白酒 *6
Sell price :1734.0
name : 南非 利斯摩酒莊西拉紅酒2017 Lismore Syrah *6
Sell price :2040.0
name : 維地那山丘 佩果雷朵紅酒2017 Montevertine Le Pergole Torte *6
Sell price :9900.0
name : 野狼一級園夏佈利 Thomas Pico Domaine Pattes Loup Chabli *6
Sell price :4500.0
name : 意大利波吉紅酒2016 IL POGGIONE BRUNELLO DI MONTALCINO
Sell price :1980.0
===========================
data for page 2
name : Mollydooker Miss Molly 澳洲夫妻檔茉莉小姐氣泡酒2017*6
Sell price :1479.0
name : 弗萊迪 香波慕西尼一級園2017 *6
Sell price :5160.0
name : 弗萊迪 夜聖喬治一級園白酒2018 *6
Sell price :4200.0
name : 弗萊迪 莫雷聖丹尼2017 *6
Sell price :3420.0
name : 阿根廷 安第斯白馬2018 *6
Sell price :4800.0
name : 法國彭特薩克副牌2011 Chapelle e Potensac*6
Sell price :780.0
name : 法國柏菲副牌 2014 Aromes De Pavie*6
Sell price :4320.0
name : 義大利麓鵲紅酒2018 Tenuta Luce IGT
Sell price :4680.0
name : 德頌 熱夫雷 香貝丹GC紅酒2015 Yann Durieux GC *6
Sell price :15780.0
name : 美國尤利西斯赤霞珠2015 Ulysses Napa valley*6
Sell price :8220.0
name : 法國凱歌莊園紅酒2003 Chateau Hosanna *6
Sell price :7680.0
name : 拉旺正牌2008年 *6
Sell price :8580.0
name : 法國萊昂丘杜查斯甜白酒1994年Moulin Touchais*6
Sell price :3132.0
name : 法國萊昂丘杜查斯甜白酒1985年 Moulin Touchais*6
Sell price :3550.0
name : 法國萊昂丘杜查斯甜白酒1982年 Moulin Touchais*6
Sell price :3800.0
name : Branaire Ducru Duluc 法國周伯通副牌2009 *6
Sell price :1470.0
name : Chateau Clerc Milon法國雙公正牌2017 *6
Sell price :5160.0
name : 2017年智利普古拉旱地 Polkura Secano *6
Sell price :2766.0
name : 紐西蘭雲霧灣-蒂蔻蔻長相思2016 Cloudy Bay Te koko SB *6
Sell price :2700.0
name : ERRAZURIZ Don Maxi Founder 2016 伊拉蘇莊主珍藏 *6
Sell price :3420.0
name : 2019 西班牙奧托老藤精選 Aalto PS *6
Sell price :5742.0
name : 法國彭特康耐正牌2012 Chateau Pontet-Canet *6
Sell price :4920.0
name : (375ml)法國太保副牌2016年 Connetable Talbot *12
Sell price :1980.0
name : 法國捷高正牌(美人魚)1996 CHA. Giscours Margaux *6
Sell price :4380.0
name : 紐西蘭新天地酒莊黑皮諾2018 Ata Rangi Pinot Noir*6
Sell price :3180.0
name : 澳洲泰勒氏Taylors St.Andrews Cab Sau *6
Sell price :2528.0
name : 彭特康耐2009年 *6
Sell price :12600.0
name : 南非漢密爾頓酒莊 霞多麗白酒2021 Hamilton Russell Vineyards *6
Sell price :2297.0
name : sena2013 *6
Sell price :6000.0
name : 克萊登山布魯克曼梅洛Clarendon Brookman Merlot2007 *6
Sell price :2142.0
name : 法國傳奇2015 Legende Bordeaux *6
Sell price :300.0
name : 法國奧斯葉'紅A'紅酒2015 Aussieres Rouge 6
Sell price :288.0
name : 澳洲克萊登山星光園稀世珍藏2008 Clarendon Hills Astralis *6
Sell price :10320.0
name : 意大利波特嘉粉紅玫瑰氣酒 BOTTEGA ROSE MANZONI MOSCATO *6
Sell price :1560.0
name : Branaire Ducru Duluc 法國周伯通副牌2012 *6
Sell price :1350.0
name : 法国靚次伯副牌2004 CH.Haut-Bages Averous *6
Sell price :2400.0
name : 布根地律動阿羅克斯-高登村 *6
Sell price :2580.0
name : 法國雄驢 歌海娜紅酒 *12
Sell price :2083.0
name : Chateau D'Armailhac 法國單公正牌2016 *6
Sell price :3300.0
name : 楯野川 大吟釀梅酒 *6
Sell price :3060.0
name : (375ml)法國奧利弗正牌紅酒2015 Chateau Olivier
Sell price :4329.0
name : Le Clarence de Haut Brion 法國紅顏容副牌2015 *6
Sell price :7980.0
name : 義大利嘉雅 巴巴萊斯科紅酒2003 Gaja Barbaresco DOCG *6
Sell price :15480.0
name : 法國彭特康耐正牌2012 Chateau Pontet-Canet*6
Sell price :4920.0
name : Chateau Leoville Barton 法國巴頓正牌2012 *6
Sell price :4380.0
name : 法國拉格蘭吉正牌2009 CHA. Lagrange *6
Sell price :3758.0
name : 法國龍船正牌2016 Chateau Beychevelle*6
Sell price :6540.0
name : 法國龍船正牌2016 Chateau Beychevelle*12
Sell price :12720.0
name : 人頭馬精選CellarN16 REMY MARTIN Prime Cellar C16 *6
Sell price :3000.0
name : 人頭馬精選CellarN16 REMY MARTIN Prime Cellar C16 *12
Sell price :5880.0
===========================
data for page 3
name : 羅伯特黑皮諾精選2014 Robert Mondavi Reserve Pinot Noir *6
Sell price :4685.0
name : 意大利羅密歐BDM紅酒2015年 *6
Sell price :2760.0
name : 意大利羅密歐BDM紅酒2015年 *12
Sell price :5280.0
name : 法国靚次伯正牌2010 CH.Lynch Bages *6
Sell price :8700.0
name : 法國雄獅高樂士莊園2010 Clos Du Marouis *6
Sell price :3240.0
name : 法國利是金正牌2017 Chateau Lascombes *6
Sell price :3540.0
name : 法國拉拉貢紅酒2014年 Chateau La Lagune *6
Sell price :2746.0
name : 法國魯臣世家正牌2008 Chateau Rauzan-Segla *6
Sell price :4224.0
name : Chateau Pichon Longueville 法國女碧崗正牌2009 *6
Sell price :10860.0
name : 法國 布魯諾百漾香檳 Bruno Paillard Premiere Cuvee Brut *6
Sell price :2798.0
name : Opus One 美國作品一號2014 *6
Sell price :18300.0
name : 法國靚次伯正牌白酒 Chateau Lynch-Bages 2020 *6
Sell price :4020.0
name : Chateau Calon Segur 法國卡龍沙高(情人的心)2017 *6
Sell price :4910.0
name : 法國玫瑰正牌2014 Chateau Montrose Saint-Estephe *6
Sell price :7740.0
name : 法國瑪歌副牌2018 Pavillon Rouge Chateau Margaux
Sell price :12000.0
name : 法国靚次伯正牌2013 CH.Lynch Bages *12
Sell price :12000.0
name : 布根地樂弗雷(雙雞)馬貢維赫杰白酒2018 leflaive Macon Verze
Sell price :4176.0
name : 澳洲露紋酒莊 藝術霞多麗白酒 2018 Leeuwin Estate Art Series *6
Sell price :3550.0
name : 義大利皮奧頂峰 巴巴萊斯科紅酒2013 Pio Cesare Barbaresco *6
Sell price :5220.0
name : 伍傑雷酒莊 普里尼-蒙哈榭白酒2017 Domaine De La Vougeraie PM *6
Sell price :4800.0
name : 弗朗索瓦-卡瑞濃酒莊 普里尼 -蒙哈榭白酒2018 Francois Carillon PM *6
Sell price :3915.0
name : 弗朗索瓦-卡瑞濃酒莊 夏山-蒙哈榭白酒2019 Francois Carillon CM *6
Sell price :4646.0
name : 意大利西西里島 星球酒莊 卡里坎特白酒2019 Planeta Etna *6
Sell price :1723.0
name : 美國索諾瑪 克雷瑪酒莊俄羅斯河谷黑皮諾2017 La Crema *6
Sell price :2715.0
name : 美國克雷瑪酒莊 索諾瑪海岸霞多麗白酒2018 La Crema Sonoma *6
Sell price :1879.0
name : 拉芳公爵繼承者 馬貢霞多麗白酒2019 Heritiers Macon Chardonnay *6
Sell price :1983.0
name : 1公升威雀金冠威士忌 The Famous Grouse *3
Sell price :285.0
name : 1公升威雀金冠威士忌 The Famous Grouse *6
Sell price :540.0
name : 1公升威雀金冠威士忌 The Famous Grouse*12
Sell price :1020.0
name : 路易王妃首席香檳(禮盒) Louis Roederer Collection 242 NV*6
Sell price :2349.0
name : 路易王妃首席香檳(單支) Louis Roederer Collection 242 NV*6
Sell price :2245.0
name : 美國 斯柏斯活得旗艦酒2014 Spottswoode Estate CS *6
Sell price :11100.0
name : 美國羅伯特赤霞珠2019 Robert Mondavi Cab Sau *6
Sell price :2820.0
name : 法國泰廷爵 旗艦款白中白香檳 2011 Taittinger Champagne *6
Sell price :9900.0
name : 意大利碧安仙蒂 布魯奈羅紅酒2012 Biondi-Santi BDM Annata *6
Sell price :9480.0
name : 德頌 夜NC紅酒2018 Yann Durieux Night Cost *6
Sell price :4800.0
name : 德頌橋樑 橋LP紅酒2018 Yann Durieux Les Ponts *6
Sell price :4560.0
name : 德頌 黑皮諾BP紅酒2018 Yann Durieux Black Pinot *6
Sell price :5880.0
name : 德頌 橋頭TDP紅酒2017 Yann Durieux TDP *6
Sell price :8640.0
name : 德頌橋樑 PV紅酒2018 Yann Durieux Recrue des Sens PV *6
Sell price :8340.0
name : 斯繽尼塔 加萊迪巴羅洛紅酒2016 LA Spinetta Barolo Vigneto *6
Sell price :4200.0
name : 德頌 BT紅酒2017 Yann Durieux BT *6
Sell price :7560.0
name : 布根地潘芙洛酒莊 蓋特一級園Domaine JM & Hugues Pavelot *6
Sell price :3236.0
name : 凱慕恩 聖羅曼白酒2017 Meo-Camuzet Saint-Romain Blanc *6
Sell price :3445.0
name : 積架多利安單一園黃金級白酒2019 E.Guigal Condrieu La Doriane *6
Sell price :5100.0
name : 美國歐林斯威夫特 大千世界混釀紅酒 Orin Swift Abstract Red *6
Sell price :2923.0
name : 麓鵲酒莊布魯奈羅紅酒2016 Luce della vite BDM *6
Sell price :4920.0
name : Chateau Leoville Barton 法國巴頓正牌2009 *6
Sell price :7020.0
name : 阿根廷 敵人酒莊波爾多混釀精選紅酒2017 El Enemigo *6
Sell price :3180.0
name : 阿根廷 敵人酒莊西碧諾單一園紅酒2017 El Enemigo El Cepillo *6
Sell price :4800.0
===========================so on..