beautiful soup find_all skips a class index if data is not inside a div

Question

im trying to scrape data from a website to understand my prob here is some sample

1st iteration

<span class="lot-details-desc right">$7,344 USD
                        </span>
<span class="lot-details-desc right">Automatic
                        </span>
<span class="lot-details-desc right">Mercedes
                        </span>

2nd iteration

<span class="lot-details-desc right">$6000 USD
                        </span>
<span class="lot-details-desc right">     #NO DATA HERE
                        </span>
<span class="lot-details-desc right">Mercedes
                        </span>

#IN A LOOP while retrieving using beautiful soup

price = soup.find_all("span", {"class": "lot-details-desc right"})[0].get_text()           
  print(price)    
  trans = soup.find_all("span", {"class": "lot-details-desc right"})[1].get_text()           
  print(trans)
  name = soup.find_all("span", {"class": "lot-details-desc right"})[2].get_text()     
  print(trans)

i get the result

1st iteration
price=$7,344 USD
trans=Automatic
name=Mercedes 
     

2nd iteration
price=$6000 USD
trans=Mercedes
name=ERRORRR( out of bound cuz this one find_all indicates only 0 and 1 index instead of 0 1 2)

any suggestions would be appreciated

Answer 1

The data on that site is loaded dynamically via JavaScript. You can use requests module to get the data from their API directly:

import re
import json
import requests


url = 'https://www.copart.com/lot/25831510/'
data_url = 'https://www.copart.com/public/data/lotdetails/solr/{lot_id}'
headers = {'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:77.0) Gecko/20100101 Firefox/77.0'}

lot_id = re.search(r'lot/(\d+)', url).group(1)


with requests.session() as s:
    s.get(url, headers=headers).text # load cookies
    data = s.get(data_url.format(lot_id=lot_id), headers=headers).json()

    # ucomment this to see all data:
    # print(json.dumps(data, indent=4))

    name = data['data']['lotDetails']['mkn']
    trans = data['data']['lotDetails']['tsmn']
    price = data['data']['lotDetails']['la']

    print('Name={} Trans={} Price={}'.format(name, trans, price))

Prints:

Name=TOYOTA Trans=AUTOMATIC Price=7344.0