Search code examples
pythonselenium-webdriverxpathcss-selectorswebdriver

How to get all links with python selenium in a tbody

Here's my python script that allows to search the https://whatsmyname.app/ site and then retrieve the links with this:

link = driver.find_elements(by=By.XPATH, value="/html/body/div/div/div[3]/div[2]/div/div[2]/table/tbody")
for row in link:
    href = row.get_attribute("href")
    print(href)

However, it doesn't work. I thought of another alternative, let me explain. When it fetches the <tbody>, I want it to be able to go to all the class td's and give its link. I've tried several things like this:

for row in driver.find_elements(by=By.XPATH, value="/html/body/div/div/div[3]/div[2]/div/div[2]/table/tbody"):
    print(row.get_attribute("class"))

The link of the website : https://whatsmyname.app/

But it didn't work again. It prints nothing. Could someone help me?

Solution

  • You were close enough. The locator strategy which you have used:

    link = driver.find_elements(by=By.XPATH, value="/html/body/div/div/div[3]/div[2]/div/div[2]/table/tbody")

    identifies only one single element, i.e. the only <tbody> element. Now, the <tbody> element neither have the href attribute nor the class attribute. Hence nothing is printed.

    Solution

    The href attributes are within the descendent <a> tags of the forth <td> within their ancestor <tr> tags. So your effective line of code will be:

    • Using CSS_SELECTOR:

      link = driver.find_elements(by=By.CSS_SELECTOR, value="table#collectiontable > tbody tr td:nth-child(4)")
for row in link:
  href = row.get_attribute("href")
  print(href)

    • Using XPATH:

      link = driver.find_elements(by=By.XPATH, value="//table[@id='collectiontable']/tbody//tr//following::td[4]/a")
for row in link:
  href = row.get_attribute("href")
  print(href)

    Inducing WebDriverWait for visibility_of_all_elements_located() your optimized code block can be:

    driver.get("https://whatsmyname.app/")
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//textarea[@id='targetUsername']"))).send_keys("Look")
driver.find_element(By.XPATH, "//button[@class='btn btn-success']/i").click()
time.sleep(10)
print([my_elem.get_attribute("href") for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "table#collectiontable > tbody tr td:nth-child(4) > a")))])
print([my_elem.get_attribute("href") for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//table[@id='collectiontable']/tbody//tr//following::td[4]/a")))])
driver.quit()

    Note : You have to add the following imports :

    from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

    Console Output:

    ['http://1001mem.ru/Look', 'https://forum.3dnews.tech/member.php?username=Look', 'https://3dtoday.ru/blogs/Look', 'https://www.7cups.com/@Look', 'https://about.me/Look', 'https://uk.advfn.com/forum/profile/Look', 'https://akniga.org/profile/Look', 'https://projecthub.arduino.cc/Look', 'https://ask.fm/Look', 'https://au.ru/user/Look/', 'https://community.avid.com/members/Look/default.aspx', 'https://forums.babypips.com/u/Look/summary', 'https://bandcamp.com/Look', 'https://www.bandlab.com/Look', 'https://www.babyblog.ru/user/Look', 'https://www.behance.net/Look', 'https://www.bigo.tv/user/Look', 'https://www.bitchute.com/channel/Look/', 'https://blip.fm/Look', 'http://look.blogspot.com/', 'http://bodyspace.bodybuilding.com/Look/', 'https://www.buymeacoffee.com/Look', 'https://www.buzzfeed.com/Look', 'https://cafecito.app/Look', 'https://calendly.com/Look', 'https://look.carbonmade.com/', 'https://career.habr.com/Look', 'https://www.castingcall.club/Look', 'https://www.cda.pl/Look', 'https://forum.cfx.re/u/Look/summary', 'https://www.championat.com/user/Look/', 'https://profile.cheezburger.com/Look', 'https://www.chess.com/member/Look', 'https://chomikuj.pl/Look/', 'https://codeforces.com/profile/Look', 'https://coderwall.com/Look/', 'https://www.coroflot.com/Look', 'https://www.cracked.com/members/Look', 'https://crowdin.com/profile/Look', 'https://cults3d.com/en/users/Look/creations', 'https://darudar.org/users/Look/', 'https://demotywatory.pl/user/Look', 'https://www.designspiration.com/Look/', 'https://dev.to/Look', 'https://www.discogs.com/user/Look', 'https://www.donationalerts.com/r/Look', 'https://www.duolingo.com/profile/Look', 'https://www.ebay.com/str/Look', 'https://f3.cool/Look', 'https://www.fabswingers.com/profile/Look']