Even the python code seems correct but attribute error occur, no text scraped

Question

When I was using BeautifulSoup to scrape listing product name and price, the similar code worked on other website. But when running in this website, soup.findAll attributes are there but no text scraped, AttributeError occurred. Is anyone can help to take look the code and website inspect?

I checked and ran many times, the same issue remained

Codes are here:

url = 'https://shopee.co.id/Handphone-Aksesoris-cat.40'
re = requests.get(url,headers=headers)
print(str(re.status_code))
soup = BeautifulSoup(re.text, "html.parser")
for el in soup.findAll('div', attrs={"class": "collection-card_collecton-title"}):
    name = el.get.text()
    print(name)

AttributeError: 'NoneType' object has no attribute 'text'

Answer 1

You are missing an i in the class name. However, content is dynamically loaded from API call (which is why you can't find it in your call where js doesn't run and so this next call to update DOM doesn't occur); you can find in network tab. It returns json.

import requests

r = requests.get('https://shopee.co.id/api/v2/custom_collection/get?category_id=40&platform=0').json()
titles = [i['collection_title'] for i in r['collections'][0]['list_popular_collection']]
print(titles)

---

---

Prices as well:

import requests

r = requests.get('https://shopee.co.id/api/v2/custom_collection/get?category_id=40&platform=0').json()
titles,prices =zip(*[(i['collection_title'], i['price']) for i in r['collections'][0]['list_popular_collection']])
print(titles,prices)