Webscrape URL link when there's no href? With BeautifulSoup

Question

I'm trying to scrape each Product's Brand, Name, Image URL from: https://www.mecca.com.au/skin-care/

I'm having trouble scraping the Image URL as there's no href

For example one of the product's page: https://www.mecca.com.au/tatcha/indigo-cleansing-balm/I-059912.html?cgpath=skincare

<picture class="css-1azgcry-container" id="image-reponsive-container" data-testid="imageReponsiveContainer">
  <img class="" title="Tatcha - Indigo Cleansing Balm" alt="Tatcha - Indigo   Cleansing Balm" src="https://www.mecca.com.au/on/demandware.static/-/Sites-mecca-online-catalog/default/dw0a0707d8/product/tatcha/hr/i-059912-indigo-cleansing-balm-7-1-940.jpg" data-testid="imageReponsiveImg"></picture>

I would like to extract just the Image URL following src eg. here:

src="https://www.mecca.com.au/on/demandware.static/-/Sites-mecca-online-catalog/default/dw0a0707d8/product/tatcha/hr/i-059912-indigo-cleansing-balm-7-1-940.jpg"

Here is my code:

import requests
from bs4 import BeautifulSoup

url = "https://www.mecca.com.au/skin-care/"
params = {"start": "0", "sz": "36", "format": "ajax"}
for params["start"] in range(0, 36 * 5, 36):
    productlinks = []
    cataloguelist = []
    soup = BeautifulSoup(requests.get('https://www.mecca.com.au/skin-care/', params=params).content, "html.parser")
    products = soup.find_all('div', class_="grid-product-info")

    for item in products:
        for link in item.find_all('a', href=True):
            productlinks.append(url + link['href'])

    for link in productlinks:
        response = requests.get(link)

        soup = BeautifulSoup(response.content, "html.parser")
        brand = soup.find('a', class_='css-1p371np-size5-size5-sansSerif-sansSerif-brandNameLink').text
        name = soup.find('span', class_='css-1noela6-paragraph-paragraph-sansSerif-sansSerif-productName').text
### This is where I'm struggling, I tried 'picture', class_='css-1azgcry-container' as well
        image = soup.find('picture', attrs={'css-1azgcry-container', 'img', 'alt', 'src'})
        
print(brand, name, image)

Thanks for any help!

Answer 1

In your code, if you will print(image), you will see the HTML:

<picture class="css-1azgcry-container" data-testid="imageReponsiveContainer" id="image-reponsive-container"><img alt="MECCA COSMETICA - Renewing Gel Cleanser" class="" data-testid="imageReponsiveImg" src="https://www.mecca.com.au/on/demandware.static/-/Sites-mecca-online-catalog/default/dwf0d61029/product/mecca/hr/i-061159-renewing-gel-cleanser-1-940.jpg" title="MECCA COSMETICA - Renewing Gel Cleanser"/></picture>

Which basically means you need to dig one level deeper. The best way to do so would be to get the id (id="image-reponsive-container) since Id's are unique.

Instead of:

image = soup.find('picture', attrs={'css-1azgcry-container', 'img', 'alt', 'src'})

Use:

image = soup.find(id='image-reponsive-container').find('img').get('src')

Or even better, a CSS selector:

image = soup.select_one('#image-reponsive-container > img')['src']