Search code examples
pythonseleniumweb-scrapingxpathclassname

Error finding element with selenium python of transfer market website?(tried with xpath,class name etc) but nothing works

Hi Dear Programmers and webscrapers.I have tried so much times to locate the element and then press it but it always gives me error in selenium.I have treid using XPATH,class name and id of the element is not present so I tried with multiple things but nothing works and returns me an error. This is the website link that I want to scrap. Website to scrap Here is the element that I wanted to scrap. This is Image showing Element I wanted to locate

This element had class selector-title so I tried with class name and also with xpath of this element and with its sub elements also but unfortunately was not able to locate it. Here is the code that i used.This code is working for other functions like I have located search box and have searched pressed button it works.Right.Where it dosent work is the element I have been trying it to work.Please let me know what mistake is from mine side so this can be workable or I can locate it and then click on it. Actually this I have to do for web scraping the data of all the players in a club or in a league so for that I have to select the league of country first then I will be able to scrap the data so its very much important for me to actually be able to locate and click the marked element.Please help me if any one of you can.Thanks.

The picture showing inspect of the browser

XPATH of the marked element

//*[@id="main"]/header/div[3]/tm-quick-select-bar//div/tm-quick-select[1]/div/div

Here is my code

import os
from selenium import webdriver
from selenium.webdriver.common.by import By
import time
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait as wait

os.environ['PATH'] += r"C:/ChromeDriver"
browser = webdriver.Chrome()
browser.get("https://www.transfermarkt.us/")
element = browser.find_element(By.XPATH, '//*[@id="schnellsuche"]/input[1]')
element.send_keys("J1 League")
time.sleep(5)

# element2 = browser.find_element(By.XPATH, '//*[@id="main"]/header/div[3]/tm-quick-select-bar//div/tm-quick-select[1]/div/div[1]/strong/text()')
# element2.click()

button = wait(browser, 10).until(EC.presence_of_element_located(
    (By.CLASS_NAME, 'selector-title')))
button.click()

time.sleep(10)

The error I get is the browser instantly closes as soon as it reaches the last element target element to click

DevTools listening on ws://127.0.0.1:61330/devtools/browser/d6f23f8f-6bea-41dd-8500-08f85c9cb502
[18424:9224:0523/200949.930:ERROR:device_event_log_impl.cc(214)] [20:09:49.929] USB: usb_device_handle_win.cc:1049 Failed to read descriptor from node connection: A device attached to the system is not functioning. (0x1F)
[18424:9224:0523/200949.936:ERROR:device_event_log_impl.cc(214)] [20:09:49.935] USB: usb_device_handle_win.cc:1049 Failed to read descriptor from node connection: A device attached to the system is not functioning. (0x1F)
[18424:9224:0523/200949.951:ERROR:device_event_log_impl.cc(214)] [20:09:49.951] Bluetooth: bluetooth_adapter_winrt.cc:1075 Getting Default Adapter failed.
Traceback (most recent call last):
  File "d:\WeBScraping\FreecodecampWebScrapingTutorial\selenium_tutorial_freecodecamp\scrapingtarget.py", line 18, in <module>
    button = wait(browser, 10).until(EC.presence_of_element_located(
  File "C:\Users\bilal\AppData\Roaming\Python\Python310\site-packages\selenium\webdriver\support\wait.py", line 89, in until
    raise TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message:
Stacktrace:
Backtrace:
        Ordinal0 [0x00E1B8F3+2406643]
        Ordinal0 [0x00DAAF31+1945393]
        Ordinal0 [0x00C9C748+837448]
        Ordinal0 [0x00CC92E0+1020640]
        Ordinal0 [0x00CC957B+1021307]
        Ordinal0 [0x00CF6372+1205106]
        Ordinal0 [0x00CE42C4+1131204]
        Ordinal0 [0x00CF4682+1197698]
        Ordinal0 [0x00CE4096+1130646]
        Ordinal0 [0x00CBE636+976438]
        Ordinal0 [0x00CBF546+980294]
        GetHandleVerifier [0x01089612+2498066]
        GetHandleVerifier [0x0107C920+2445600]
        GetHandleVerifier [0x00EB4F2A+579370]
        GetHandleVerifier [0x00EB3D36+574774]
        Ordinal0 [0x00DB1C0B+1973259]
        Ordinal0 [0x00DB6688+1992328]
        Ordinal0 [0x00DB6775+1992565]
        Ordinal0 [0x00DBF8D1+2029777]
        BaseThreadInitThunk [0x76CDFA29+25]
        RtlGetAppContainerNamedObjectPath [0x76F57A7E+286]
        RtlGetAppContainerNamedObjectPath [0x76F57A4E+238]
Solution

  • Not sure if I understood you correctly, but the code below return the following text "United States".

    import time
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait as wait

driver = webdriver.Chrome(options=options, desired_capabilities=capabilities)

# open url:
driver.get('https://www.transfermarkt.us/')

# switch to cookies frame
wait(driver, 10).until(EC.frame_to_be_available_and_switch_to_it((By.ID, "sp_message_iframe_575847")))
time.sleep(3)

# click "Accept" cookies button
driver.find_element(By.XPATH, '//button[@title="ACCEPT ALL"]').click()
time.sleep(3)

# go back to main frame
driver.switch_to.default_content()

# here's the trick, what you are looking for is inside a "shadow-root" DOM so to access it you need to execute the script and then use CSS selector, I don't think XPATH works here:
country = driver.execute_script("return document.querySelector('tm-quick-select-bar').shadowRoot.querySelector('div > tm-quick-select:nth-child(2) > div > div > strong')")

# print
print(country.text)

# close the driver
driver.close()