I am super new to beautifulsoup, I have done tons of online videos and now I am adventuring to my first project.
Anyway, my goal is to scrape the location of https://www.mastermindtoys.com/apps/store/find-a-store. All the locations are under one class "clearfix large-container". I am wondering how do I pull out the information of the address from all "address-sec". "address-sec" being the class that is under "clearfix large-container". If anyone has a video that goes inline or at advice that would be great!
The data you see is loaded from external source, so beautifulsoup doesn't see it. But you can simulate Ajax requests with requests module to load it:
import json
import requests
from bs4 import BeautifulSoup
url = "https://bopis.mastermindtoys.com/ajax/getstorelocation.php"
headers = {
"Referer": "https://www.mastermindtoys.com/",
}
payload = {
"doAction": "getNearStoreData",
"latitude": "",
"longitude": "",
"flagDisplayFirstRecord": "Y",
"shopifyCustomerId": "",
"selectedStoreInCookies": "",
}
data = requests.post(url, headers=headers, data=payload).json()
# uncomment this to print all data:
# print(json.dumps(data, indent=4))
for i, d in enumerate(data["details"], 1):
print(
"{:<3} {:<25} {:<30} {:<30}".format(
i, d["name"], d["address1"], d["address2"]
)
)
Prints:
1 Abbotsford 32470 South Fraser Way Clearbrook Town Square
2 Langley 20085 Langley Bypass Langley Centre
3 Coquitlam 3000 Lougheed Hwy Westwood Mall
4 Tsawwassen 5000 Canoe Pass Way Tsawwassen Mills
5 Langford 2945 Jacklin Road Westshore Town Centre
6 Prince George 3034 Recplace Drive River Point Landing
7 Westhills 600 Stewart Green SW Westhills Town Centre
8 Shawnessy 16061 MacLeod Trail SE Boulevard Shopping Centre
9 McKenzie Towne 4307 130 Ave SE South Trail Crossing
10 Country Hills 450 Country Hills Blvd NE Country Hills Town Centre
11 Lethbridge 1250 2A Avenue North
12 Red Deer 5250 22nd Street
13 Grande Prairie 11521 Westgate Drive Westgate Centre
14 Windermere 5236 Windermere Boulevard NW Currents of Windermere
15 Terra Losa 9752 170 Street Terra Losa Centre
16 South Edmonton 2004 99th Street NW
17 St. Albert 398 St. Albert Trail Mission Hill Plaza
18 Sherwood Park 222 Baseline Road Sherwood Park Baseline Village Plaza
19 Medicine Hat 1840 Strachan Road SE
20 Saskatoon South 3011 Clarence Avenue South Unit 170
21 Preston Crossing 1718 Preston Avenue North
22 Regina East 2820 East Quance Street
23 Kenaston 1655 Kenaston Boulevard
24 Kildonan 1561 Regent Avenue West Kildonan Mall
25 Sudbury 1332 Kingsway Boulevard
26 Windsor 3175 Howard Avenue Roundhouse Centre
27 Owen Sound 1969 16th Street East
28 London North 685 Fanshawe Park Road West
29 London South 3120 Wonderland Road South Wonderland Plaza
30 Waterloo 405 King Street North
31 Barrie 488 Bayfield Street North Supercentre
32 Cambridge 612 Hespeler Road
33 Guelph 370 Stone Road West Stone Square Centre
34 Milton 1079 Maple Avenue
35 Newmarket 18267 Yonge Street
36 Aurora 650 Wellington Street East Aurora Gateway Centre
37 Ancaster 821 Golf Links Road Meadowlands Entertainment Centre
38 Vaughan 16 Famous Avenue Colossus Centre
39 Hurontario 26 Eglinton Avenue West
40 Burlington 1035 Brant Street
41 Oakville 1011 Upper Middle Road East Upper Oakville Shopping Centre
42 Richmond Hill 9350 Yonge Street
43 Mississauga South 970 Southdown Road Clarkson Crossing Centre
44 Etobicoke 4242 Dundas Street West Kingsway Mills Shopping Centre
45 Hamilton 2180 Rymal Road East
46 Bayview Village 2901 Bayview Avenue Bayview Village Shopping Centre
47 Yonge 3350 Yonge Street
48 Markham 8555 McCowan Rd
49 Mount Pleasant 639 Mount Pleasant Road
50 Rosedale 1133 Yonge Street
51 The Beach 2134 Queen Street East
52 Pickering 1355 Kingston Road Pickering Town Centre
53 Ajax 170 Kingston Road East
54 North Oshawa 991 Taunton Road East
55 St. Catharines 210 Glendale Avenue
56 Peterborough 995 Lansdowne Street West
57 Kanata 145 Roland Michener Drive Kanata Centrum Plaza
58 Kingston 2511 Princess Street
59 Barrhaven 71 Marketplace Avenue Chapman Mills Marketplace
60 St. Laurent 1091 St Laurent Boulevard
61 Orléans 1315 Place d'Orleans Drive Place d'Orléans
62 Fredericton 38 Trinity Avenue
63 Saint John 70 Consumers Drive Westmorland Shopping Centre
64 Moncton 84 Wyse Street Mapleton Centre
65 Halifax 201 Chain Lake Drive
66 Dartmouth 29 Countryview Drive Dartmouth Crossing
67 Sydney 20 Sydney Port Access Road
68 St. John's North 50 White Rose Drive