So here's my code first:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.options import Options
import time
from fake_useragent import UserAgent
import random
ua = UserAgent()
options = Options()
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--blink-settings=imagesEnabled=false')
chrome_options.add_argument('--headless')
chrome_options.add_argument(f'user-agent={ua.random}')
driver = webdriver.Chrome(options=options, chrome_options=chrome_options)
driver.maximize_window()
url = "https://magiceden.io/marketplace/hasuki"
driver.get(url)
element = driver.find_element(By.CSS_SELECTOR, "#content > div.tw-w-full.tw-py-0.sm\:tw-mt-0 > div.tw-flex.tw-relative > div.tw-flex-auto.tw-max-w-full.tw-pt-0 > div.tw-flex.tw-items-center.md\:tw-justify-between.tw-gap-2.md\:tw-gap-4.md\:tw-sticky.tw-top-\[133px\].tw-bg-gray-100.tw-z-10.tw-flex-wrap.tw-p-5 > div.tw-flex.tw-flex-grow.tw-justify-center.tw-gap-x-2 > button > span:nth-child(4)")
print(f"The current instant sell price is {element.text}")
When I run it, I get weird long error, that ends with:
Backtrace:
(No symbol) [0x00806643]
(No symbol) [0x0079BE21]
(No symbol) [0x0069DA9D]
(No symbol) [0x006D1342]
(No symbol) [0x006D147B]
(No symbol) [0x00708DC2]
(No symbol) [0x006EFDC4]
(No symbol) [0x00706B09]
(No symbol) [0x006EFB76]
(No symbol) [0x006C49C1]
(No symbol) [0x006C5E5D]
GetHandleVerifier [0x00A7A142+2497106]
GetHandleVerifier [0x00AA85D3+2686691]
GetHandleVerifier [0x00AABB9C+2700460]
GetHandleVerifier [0x008B3B10+635936]
(No symbol) [0x007A4A1F]
(No symbol) [0x007AA418]
(No symbol) [0x007AA505]
(No symbol) [0x007B508B]
BaseThreadInitThunk [0x75EB00F9+25]
RtlGetAppContainerNamedObjectPath [0x77A27BBE+286]
RtlGetAppContainerNamedObjectPath [0x77A27B8E+238]
BUT if I comment out "chrome_options.add_argument('--headless')", my code works perfectly fine. What's the issue here? I suppose the problem is that website doesn't let me use headless mode, how can I solve this?
I want my program to run in headless mode, but I get restricted either by the website or chrome browser.
Basically, you are seeing a NoSuchElementException
Full stacktrace:
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":"#content > div.tw-w-full.tw-py-0.sm\:tw-mt-0 > div.tw-flex.tw-relative > div.tw-flex-auto.tw-max-w-full.tw-pt-0 > div.tw-flex.tw-items-center.md\:tw-justify-between.tw-gap-2.md\:tw-gap-4.md\:tw-sticky.tw-top-\[133px\].tw-bg-gray-100.tw-z-10.tw-flex-wrap.tw-p-5 > div.tw-flex.tw-flex-grow.tw-justify-center.tw-gap-x-2 > button > span:nth-child(4)"}
(Session info: headless chrome=109.0.5414.75)
Stacktrace:
Backtrace:
(No symbol) [0x00E04AD3]
(No symbol) [0x00D99211]
(No symbol) [0x00C8D9CD]
(No symbol) [0x00CC1102]
(No symbol) [0x00CC123B]
(No symbol) [0x00CF8A72]
(No symbol) [0x00CDFB84]
(No symbol) [0x00CF67B9]
(No symbol) [0x00CDF936]
(No symbol) [0x00CB4788]
(No symbol) [0x00CB5C1D]
GetHandleVerifier [0x01079342+2502274]
GetHandleVerifier [0x010A7959+2692249]
GetHandleVerifier [0x010AABDC+2705180]
GetHandleVerifier [0x00EB3480+643008]
(No symbol) [0x00DA208F]
(No symbol) [0x00DA7AB8]
(No symbol) [0x00DA7BA5]
(No symbol) [0x00DB273B]
BaseThreadInitThunk [0x76D9FA29+25]
RtlGetAppContainerNamedObjectPath [0x77B37A4E+286]
RtlGetAppContainerNamedObjectPath [0x77B37A1E+238]
The desired element is a dynamic element, so ideally to print the desired text from the element you need to induce WebDriverWait for the visibility_of_element_located() and you can use the following Locator Strategy:
Using CSS_SELECTOR:
driver.get("https://magiceden.io/marketplace/hasuki")
print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "#content > div.tw-w-full.tw-py-0.sm\:tw-mt-0 > div.tw-flex.tw-relative > div.tw-flex-auto.tw-max-w-full.tw-pt-0 > div.tw-flex.tw-items-center.md\:tw-justify-between.tw-gap-2.md\:tw-gap-4.md\:tw-sticky.tw-top-\[133px\].tw-bg-gray-100.tw-z-10.tw-flex-wrap.tw-p-5 > div.tw-flex.tw-flex-grow.tw-justify-center.tw-gap-x-2 > button > span:nth-child(4)"))).text)
driver.quit()
Console Output:
0.3664
Note: You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC