Search code examples
pythonseleniumselenium-webdriverhrefgetattribute

How to extract href attribute of an element in html with Selenium in Ptyhon


I need a make a list of URLs of pictures(after that I will download it ) I do not know how to extract elements from class and from elements extract a URLs

driver.get("https://pixabay.com/en/photos/search/" + tag +"/?orientation=horizontal&size=medium")
images = [my_elem.get_attribute("href") for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//div[@class='item']/a")))] 
print("Images on Pixabay were found")

        
for x in images:
    images = driver.find_element(By.XPATH,"/html/body/div[1]/div[2]/div/div[1]/div/div[1]/div/a[1]")
    images = images.get_attribute("href")
    print(images)
    driver.get(images)
    #this is important because if I did not open a URL(Url after open is change to another.) it did not work
    sec_url = driver.current_url
    print( "Sec URL:  " +sec_url)

Solution

  • images is a list of all the <href> attributes which you have collected using:

    images = [my_elem.get_attribute("href") for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//div[@class='item']/a")))] 
    

    Moving ahead as you are iterating through the list as:

    for x in images:
    

    you need to avoid using the same variable name to identify an element and to store the <href> attribute within the loop. So you need to modify the for loop as follows:

    for x in images:
        element = driver.find_element(By.XPATH,"/html/body/div[1]/div[2]/div/div[1]/div/div[1]/div/a[1]")
        element_href = element.get_attribute("href")
        print(element_href)
        driver.get(element_href)
        sec_url = driver.current_url
        print( "Sec URL:  " +sec_url)