Search code examples
pythonhtmlseleniumhref

Finding href link with Python selenium

I am working on a project to navigate and scrape a website with selenium, python.

hemnet_path = "https://www.hemnet.se/"
driver = webdriver.Chrome(path, chrome_options=options)
driver.get(hemnet_path)

The script navigates to this site and loops over all the listings to get price, location, hyperlink etc. by loading the result elements like this:

result = driver.find_elements(By.CLASS_NAME,"sold-property-listing")

It then reads pieces of information like:

for r in result:

    try:
        location = r.find_element(By.CLASS_NAME,"sold-property-listing__location")
    except:
        location = 'Missing'

The problem that I am having is that the hyperlink-string (screenshot) is not "behaving" like the other elements.

enter image description here

I can't seem to select by class nor by CSS selector and href because I get the following error:

enter image description here

enter image description here

I have successfully used XPATH to get the link like so: enter image description here

But the problem is that when I iterate over the results the XPATH is always refering to the same ad, 2 in this case. I have considered looping over the integer in the XPATH but that turned out to be very unreliable.

I am not sure what I am missing here and why I am getting this strange error.

Any help on this would be greatly appreciated. HTML for the... element is provided below.

<li class="sold-results__normal-hit">
        <a class="hcl-card" data-target-blank="true" href="https://www.hemnet.se/salda/lagenhet-3rum-barkarbystaden-jarfalla-kommun-karlslundsvagen-26-1492088" rel="noopener" target="_blank">
  <div class="sold-property-listing">
    <div class="sold-property-listing__location">
      <h2 class="sold-property-listing__heading">
        <span class="property-icon property-icon--result"><svg width="14" height="16" viewBox="0 0 14 16" xmlns="http://www.w3.org/2000/svg"><title>Lägenhet</title><desc><span class="svg-icon__fallback-text">Lägenhet</span></desc><path class="svg-icon__shape" d="M0 1.333v13.334C0 15.403.597 16 1.333 16h4v-5.333H8V16h4c.737 0 1.333-.597 1.333-1.333V1.333C13.333.597 12.737 0 12 0H1.333C.597 0 0 .597 0 1.333" fill="#C1569D" fill-rule="evenodd"></path></svg>
</span>
        <span class="item-result-meta-attribute-is-bold item-link qa-selling-price-title">Karlslundsvägen 26</span>
      </h2>

      <div>
        <span class="hide-element">
          Lägenhet
        </span>
          <span class="item-link">
            Barkarbystaden,
</span>        Järfälla kommun
      </div>
    </div>

    <div class="sold-property-listing__size">
        <div class="clear-children">
          <div class="sold-property-listing__subheading sold-property-listing--left">
              78&nbsp;m²
            &nbsp;
              3&nbsp;rum
          </div>

            <div class="sold-property-listing__fee">
              4&nbsp;233&nbsp;kr/mån
            </div>
        </div>


    </div>

    <div class="sold-property-listing__price">
      <div class="clear-children">
        <span class="sold-property-listing__subheading sold-property-listing--left">
            Slutpris 3&nbsp;295&nbsp;000&nbsp;kr
</span>      </div>

      <div class="clear-children">
          <div class="sold-property-listing__sold-date sold-property-listing--left">
            Såld 5 november 2021
          </div>

          <div class="sold-property-listing__price-per-m2 sold-property-listing--left">
            42&nbsp;244&nbsp;kr/m²
          </div>
      </div>
    </div>

      <div class="sold-property-listing__price-change">
        ±0&nbsp;%
      </div>

    <div class="sold-property-listing__broker">
      Länsförsäkringar Fastighetsförmedling Järfälla
    </div>

      <div class="sold-property-listing__labels">
        <div class="hcl-labels-list hcl-labels-list--row-direction">
            <span class="hcl-labels-list__label-item">
              <span class="hcl-label hcl-label--feature hcl-label--on-white-background">
                Uteplats
              </span>
            </span>
            <span class="hcl-labels-list__label-item">
              <span class="hcl-label hcl-label--feature hcl-label--on-white-background">
                Hiss
              </span>
            </span>
        </div>
      </div>
  </div>

</a>
    </li>

Solution

  • There is an accept cookies button which you will have to click on first.

    Secondly, you need to target all anchor tag, with the below CSS and then scroll to each a tag to let selenium view it.

    Code :

    driver = webdriver.Chrome(driver_path)
driver.maximize_window()
#driver.implicitly_wait(30)
wait = WebDriverWait(driver, 30)

driver.get("https://www.hemnet.se/salda/bostader?location_ids%5B%5D=473241&item_types%5B%5D=bostadsratt")
try:
    cookies = wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "div.consent__button-wrapper .hcl-button--primary")))
    driver.execute_script("arguments[0].scrollIntoView(true);", cookies)
    driver.execute_script("arguments[0].click();", cookies)
except:
    print("Could not click")
    driver.quit()
    pass

all_hrefs = driver.find_elements(By.CSS_SELECTOR, "a.hcl-card")

for href in all_hrefs:
    driver.execute_script("arguments[0].scrollIntoView(true);", href)
    link = href.get_attribute('href')
    print(link)

    Imports :

    from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

    Output :

    https://www.hemnet.se/salda/lagenhet-3rum-barkarbystaden-jarfalla-kommun-karlslundsvagen-26-1492088
https://www.hemnet.se/salda/lagenhet-2rum-barkarbystaden-jarfalla-kommun-fanriksvagen-41c-1490860
https://www.hemnet.se/salda/lagenhet-2rum-barkarbystaden-jarfalla-kommun-flygfaltsvagen-19-1490628
https://www.hemnet.se/salda/lagenhet-3rum-barkarbystaden-jarfalla-kommun-gripengatan-11,-van-2-1492409
https://www.hemnet.se/salda/lagenhet-2rum-barkarby-jarfalla-kommun-karlslundsvagen-24-1488804
https://www.hemnet.se/salda/lagenhet-3rum-barkarbystaden-jarfalla-kommun-kalvshallavagen-42-1488444
https://www.hemnet.se/salda/lagenhet-3rum-barkarbystaden-jarfalla-kommun-drakengatan-5-1488052
https://www.hemnet.se/salda/lagenhet-4rum-jarfalla-barkarbystaden-jarfalla-kommun-kalvshallavagen-36-1488008
https://www.hemnet.se/salda/lagenhet-2rum-barkarbystaden-jarfalla-kommun-barkarbyvagen-34-1487737
https://www.hemnet.se/salda/lagenhet-3rum-barkarby-jarfalla-kommun-flygarvagen-31-1487734
https://www.hemnet.se/salda/lagenhet-1rum-barkarby-jarfalla-kommun-jaktvagen-2-1487278
https://www.hemnet.se/salda/lagenhet-2rum-barkarbystaden-jarfalla-kommun-viggengatan-8-1487099
https://www.hemnet.se/salda/lagenhet-5rum-jarfalla-barkarbystaden-jarfalla-kommun-karlslundsvagen-3,-4-tr-1485991
https://www.hemnet.se/salda/lagenhet-2rum-jarfalla-jarfalla-kommun-barkarbyvagen-69-1485845
https://www.hemnet.se/salda/lagenhet-3rum-barkarbystaden-jarfalla-kommun-flygfaltsvagen-23-1485339
https://www.hemnet.se/salda/lagenhet-2rum-barkarbystaden-jarfalla-kommun-barkarbyvagen-36-1485338
https://www.hemnet.se/salda/lagenhet-3rum-barkarbystaden-jarfalla-kommun-majorsvagen-16-1484118
https://www.hemnet.se/salda/lagenhet-2rum-barkarbystaden-jarfalla-kommun-parkvagen-8,-2tr,-1482755
https://www.hemnet.se/salda/lagenhet-3rum-barkarby-jarfalla-kommun-flygarvagen-43-1482418
https://www.hemnet.se/salda/lagenhet-2rum-barkarbystaden-jarfalla-kommun-kalvshallavagen-32-1482219
https://www.hemnet.se/salda/lagenhet-2rum-barkarbystaden-jarfalla-kommun-barkarbyvagen-66-1481399
https://www.hemnet.se/salda/lagenhet-4rum-barkarbystaden-jarfalla-kommun-barkarbyvagen-34-1480551
https://www.hemnet.se/salda/lagenhet-2rum-barkarbystaden-jarfalla-kommun-parkvagen-8-1479755
https://www.hemnet.se/salda/lagenhet-3,5rum-jarfalla-barkarbystaden-jarfalla-kommun-flygfaltsvagen-19-1478682
https://www.hemnet.se/salda/lagenhet-2rum-barkarbystaden-jarfalla-kommun-kalvshallavagen-30-1477736
https://www.hemnet.se/salda/lagenhet-4rum-barkarby-jarfalla-kommun-skalbyvagen-17-1477210
https://www.hemnet.se/salda/lagenhet-2rum-jarfalla-jarfalla-kommun-viggengatan-11-1477547
https://www.hemnet.se/salda/lagenhet-2rum-jarfalla-barkarbystaden-jarfalla-kommun-majorsvagen-18-1478670
https://www.hemnet.se/salda/lagenhet-3rum-barkarbystaden-jarfalla-kommun-viggengatan-11-1476409
https://www.hemnet.se/salda/lagenhet-1rum-barkarbystaden-jarfalla-kommun-drakengatan-10-1475228
https://www.hemnet.se/salda/lagenhet-3rum-barkarbystaden-jarfalla-kommun-gripengatan-20-1474602
https://www.hemnet.se/salda/lagenhet-2rum-jarfalla-barkarbystaden-jarfalla-kommun-drakengatan-3,-2tr-1473715
https://www.hemnet.se/salda/lagenhet-2rum-barkarbystaden-jarfalla-kommun-viggengatan-10-1472189
https://www.hemnet.se/salda/lagenhet-3rum-barkarbystaden-jarfalla-kommun-karlslundsvagen-12-1472092
https://www.hemnet.se/salda/lagenhet-3rum-barkarbystaden-jarfalla-kommun-kalvshallavagen-44-van-4-4-1471582
https://www.hemnet.se/salda/lagenhet-3rum-barkarbystaden-jarfalla-kommun-karlslundsvagen-17-1471427
https://www.hemnet.se/salda/lagenhet-2rum-barkarbystaden-jarfalla-kommun-parkvagen-8,-3tr,-1471250
https://www.hemnet.se/salda/lagenhet-2rum-barkarbystaden-jarfalla-kommun-karlslundsvagen-26-utan-balkong-1471234
https://www.hemnet.se/salda/lagenhet-3rum-barkarby-jarfalla-kommun-flygarvagen-33-1471126
https://www.hemnet.se/salda/lagenhet-3rum-jarfalla-barkarbystaden-jarfalla-kommun-parkvagen-5-1471961
https://www.hemnet.se/salda/lagenhet-4rum-barkarbystaden-jarfalla-kommun-kalvshallavagen-36-1468708
https://www.hemnet.se/salda/lagenhet-2rum-barkarbystaden-jarfalla-kommun-majorsvagen-18-1467460
https://www.hemnet.se/salda/lagenhet-2rum-barkarbystaden-jarfalla-kommun-majorsvagen-16-1467443
https://www.hemnet.se/salda/lagenhet-2rum-jarfalla-barkarbystaden-jarfalla-kommun-parkvagen-8-1467324
https://www.hemnet.se/salda/lagenhet-2rum-jarfalla-barkarby-jarfalla-kommun-pilotvagen-11-1465674
https://www.hemnet.se/salda/lagenhet-3rum-barkarbystaden-jarfalla-kommun-viggengatan-12-1465518
https://www.hemnet.se/salda/lagenhet-2rum-barkarby-jarfalla-kommun-attackvagen-4-1465147
https://www.hemnet.se/salda/lagenhet-2rum-barkarbystaden-jarfalla-kommun-viggengatan-8-1463893
https://www.hemnet.se/salda/lagenhet-2rum-barkarbystaden-jarfalla-kommun-karlslundsvagen-18-1461358
https://www.hemnet.se/salda/lagenhet-2rum-barkarbystaden-jarfalla-kommun-viggengatan-10,-van-4-1460508