Search code examples
pythonhtmlselenium-webdriverscreen-scraping

Selenium - How to get attributes of href under specific parent class in Python


I'm trying to get the href attribute that is under a specific class. This is an example of the HTML: enter image description here

This class and href is repeated multiple times throughout the page

I have tried the below, however this only grabs the top one. Im trying to grab all instances of the href.

Link to the page I'm scraping:

https://www.domain.com.au/sale/toowoomba-qld-4350/?bedrooms=3-any&price=0-600000&excludeunderoffer=1

element = driver.find_element(By.XPATH, "//div[@class='slick-slide slick-active slick-current']")
links = element.find_elements(By.CSS_SELECTOR, "a[href*='https://www.domain.com.au']")
urls = []
for link in links:
    urls.append(link.get_attribute("href"))
    print(urls)

Solution

  • # Parent element
    element = driver.find_element(By.XPATH, "//div[@class='slick-slide slick-active slick-current']")
    
    # Anchor tags
    links = element .find_elements(By.CSS_SELECTOR, "a[href*='https://www.domain.com.au']")
    
    # All urls 
    urls = [l.get_attribute('href') for l in links]