How I can get the result of a price in web-scraping?

Question

I am trying to do a simple bot of web-scraping on Python to show my girlfriend when her favorite sneakers get on sale, I have seen many videos and tried a lot of things, the Title and other information about the sneaker is working, but the always results as None, I have also tried in other websites and get the same result.

import requests
from bs4 import BeautifulSoup

url = "https://www.vans.com.br/tenis-ultrarange-rapidweld-black-white/p/1003500430051U?gad_source=1"
headers = {'User-Agent': "insera aqui seu User Agent"} ## retirei meu user agent mas só pra nao postar com ele

site = requests.get(url, headers=headers)
soup = BeautifulSoup(site.content, 'html.parser')

title = soup.find('h1', class_= 'product-name').get_text()
price = soup.find('div', class_='ta-product-price__total' or 'product-price')

print(title)
print(price)

I have tried getting the price with many different things in the variable price, switch div to span and other things, but none was succeed, also tried the same thing with other websites and get the same result.

Answer 1

The data you see is loaded from external URL via JavaScript. You can try instead:

import requests

number = "1003500430051U"

url = f"https://www.vans.com.br/arezzocoocc/v2/vans/products/{number}/dynamic-product-fields?fields=DYNAMIC_FIELDS_PDP"

headers = {
    "User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:121.0) Gecko/20100101 Firefox/121.0"
}

data = requests.get(url, headers=headers).json()

for c in data["colorOptions"]:
    if c["code"] == number:
        print(c["name"], data["price"]["value"])
        break

Prints:

Tênis Ultrarange Rapidweld Black White 549.99