I'm writing a selenium program in python to get links from a website, it runs the first time but when I re-run it, it opens the website in browser properly then it gives an error
This is my code:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.service import Service
headers = {
"User-Agent":
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.19582",
"Accept-Language": "fr,fr-FR;q=0.8,en-US;q=0.5,en;q=0.3",
"Connection": "keep-alive",
"Accept-Encoding": "gzip, deflate, br",
}
edge_driver = './msedgedriver.exe'
s = Service( edge_driver )
browser = webdriver.Edge( service=s )
browser.get( "https://www.jumia.ug/always/" )
all_links = browser.find_elements( By.CLASS_NAME, "core" )
working_links = []
for l in all_links:
if l.get_attribute( "href" ) is not None:
working_links.append(l.get_attribute('href'))
print(working_links)
this is the error i got after running it every other time
C:\Users\eliHeist\PycharmProjects\webscraping\venv\Scripts\python.exe C:/Users/eliHeist/PycharmProjects/webscraping/getlinks.py
Traceback (most recent call last):
File "C:/Users/eliHeist/PycharmProjects/webscraping/getlinks.py", line 34, in <module>
if l.get_attribute( "href" ) is not None:
File "C:\Users\eliHeist\PycharmProjects\webscraping\venv\lib\site-packages\selenium\webdriver\remote\webelement.py", line 155, in get_attribute
attribute_value = self.parent.execute_script(
File "C:\Users\eliHeist\PycharmProjects\webscraping\venv\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 878, in execute_script
return self.execute(command, {
File "C:\Users\eliHeist\PycharmProjects\webscraping\venv\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 424, in execute
self.error_handler.check_response(response)
File "C:\Users\eliHeist\PycharmProjects\webscraping\venv\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 247, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.StaleElementReferenceException: Message: stale element reference: element is not attached to the page document
(Session info: MicrosoftEdge=96.0.1054.57)
Stacktrace:
Backtrace:
Microsoft::Applications::Events::ILogConfiguration::ILogConfiguration [0x00007FF60587CB62+56946]
Microsoft::Applications::Events::EventProperty::to_string [0x00007FF6054AA597+949863]
Microsoft::Applications::Events::EventProperty::to_string [0x00007FF6054AD788+962648]
Microsoft::Applications::Events::EventProperty::to_string [0x00007FF6054AE3EA+965818]
Microsoft::Applications::Events::EventProperty::to_string [0x00007FF605516D9F+1394287]
Microsoft::Applications::Events::EventProperty::to_string [0x00007FF605501546+1306134]
Microsoft::Applications::Events::EventProperty::to_string [0x00007FF605515D5D+1390125]
Microsoft::Applications::Events::EventProperty::to_string [0x00007FF6055013BF+1305743]
Microsoft::Applications::Events::EventProperty::to_string [0x00007FF6054D6AA9+1131385]
Microsoft::Applications::Events::EventProperty::to_string [0x00007FF6054D7B8F+1135711]
Microsoft::Applications::Events::EventProperty::to_string [0x00007FF60556C2E5+1743797]
Microsoft::Applications::Events::EventProperty::to_string [0x00007FF60556A771+1736769]
Microsoft::Applications::Events::EventProperty::EventProperty [0x00007FF60574B519+2521]
Microsoft::Applications::Events::EventProperty::to_string [0x00007FF6055420AE+1571198]
Microsoft::Applications::Events::ILogConfiguration::ILogConfiguration [0x00007FF60587576C+27260]
Microsoft::Applications::Events::ILogConfiguration::ILogConfiguration [0x00007FF6058750E4+25588]
Microsoft::Applications::Events::ILogConfiguration::ILogConfiguration [0x00007FF605874F36+25158]
Microsoft::Applications::Events::EventProperties::GetName [0x00007FF6057D02DC+211724]
BaseThreadInitThunk [0x00007FFF89E254E0+16]
RtlUserThreadStart [0x00007FFF8AF2485B+43]
Process finished with exit code 1
What should I be changing or not doing, the page does not refresh at all?
To print the value of the href attribute you have to induce WebDriverWait for the visibility_of_all_elements_located() and you can use either of the following Locator Strategies:
Using CSS_SELECTOR:
driver.get("https://www.jumia.ug/always/")
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button[aria-label='newsletter_popup_close-cta']"))).click()
print([my_elem.get_attribute("href") for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "a.core[data-brand='Always'][href]")))])
Using XPATH:
driver.get("https://www.jumia.ug/always/")
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//button[@aria-label='newsletter_popup_close-cta']"))).click()
print([my_elem.get_attribute("href") for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//a[@class='core' and @data-brand='Always'][@href]")))])
Console Output:
['https://www.jumia.ug/always-maxi-long-7s-15736010.html', 'https://www.jumia.ug/always-roxanne-maxi-duo-long-16s-15736015.html', 'https://www.jumia.ug/always-ultra-long-s3-8s-15736020.html', 'https://www.jumia.ug/always-roxanne-dreamz-maxi-ex-long-8s-15736016.html', 'https://www.jumia.ug/stainless-steel-thermos-flask-3-litres-silver-always-mpg51110.html', 'https://www.jumia.ug/always-roxanne-dreamz-maxi-ex-long-16s-15736017.html', 'https://www.jumia.ug/always-ultra-new-12s-vp-sup-pr-african-15736026.html', 'https://www.jumia.ug/travel-vacuum-thermo-cup-450ml-red-always-mpg59272.html', 'https://www.jumia.ug/pressing-flask-3litres-silver-always-mpg49765.html', 'https://www.jumia.ug/stainless-steel-travel-mug-silver-always-mpg69987.html', 'https://www.jumia.ug/unbreakable-2.5-litres-vaccum-flask-silver-always-mpg54323.html', 'https://www.jumia.ug/stainless-steel-vacuum-flask-pressing-3.5l-silver-always-mpg56188.html', 'https://www.jumia.ug/always-dailies-flexistyle-slim-panty-liners-breathable-flexible-with-fresh-scent-pack-of-26-10968170.html', 'https://www.jumia.ug/stainless-steel-vacuum-flask-500ml-black-always-mpg48945.html', 'https://www.jumia.ug/always-flask-pressing-colour-silver-16418821.html', 'https://www.jumia.ug/vaccum-travel-flask-450ml-blue-always-mpg73899.html', 'https://www.jumia.ug/500mls-vacuum-hot-cold-bottle-flask-sliver-always-mpg54381.html', 'https://www.jumia.ug/always-ultra-lw-14s-vp-sup-pr-african-15736025.html', 'https://www.jumia.ug/pressing-2.0-litres-unbreakable-vaccum-jar-flask-stainless-steel-always-mpg56170.html', 'https://www.jumia.ug/stainless-steel-travel-cap-black-always-mpg65745.html', 'https://www.jumia.ug/always-stainless-steel-travel-mug-0.48l-navy-blue-8224366.html', 'https://www.jumia.ug/always-thermal-flask-cup-silver-black-20572534.html', 'https://www.jumia.ug/stainless-vacuum-travel-mug-450ml-black-always-mpg56189.html', 'https://www.jumia.ug/0.5l-insulated-stainless-steel-travel-mug-colour-varies-always-mpg73677.html', 'https://www.jumia.ug/stainless-steel-travel-mug-0.5l-gold-always-mpg67096.html', 'https://www.jumia.ug/ultra-platinum-long-7-pads-always-mpg59273.html', 'https://www.jumia.ug/life-travel-flask-450ml-red-always-mpg72907.html', 'https://www.jumia.ug/always-xtra-protection-feminine-panty-liners-extra-long-92-pieces-us-4766946.html', 'https://www.jumia.ug/always-zzz-overnight-pads-for-women-size-6-with-wings-for-worry-free-nights-20ct-14336082.html', 'https://www.jumia.ug/portable-vacuum-thermos-flask-bottle-0.5ltr-silver-always-mpg48676.html', 'https://www.jumia.ug/stainless-steel-vacuum-flask-700ml-silver-always-mpg59271.html', 'https://www.jumia.ug/always-vacuum-hot-cold-bottle-flask-500mls-always-mpg46382.html', 'https://www.jumia.ug/stainlesss-steel-side-pressing-vaccum-flask-3litres-silver-always-mpg65748.html', 'https://www.jumia.ug/always-stainless-steel-thermos-flask-jug-silver-always-mpg42506.html', 'https://www.jumia.ug/always-450ml-thermal-flask-maroon-5494832.html', 'https://www.jumia.ug/always-food-flask-18949481.html', 'https://www.jumia.ug/always-portable-vacuum-1.0ltr-thermos-flask-bottle-silver-8032566.html', 'https://www.jumia.ug/always-stainless-vaccum-flask-3-litre-silver-black-always-mpg46820.html', 'https://www.jumia.ug/stainless-steel-vaccum-flask-3.5l-silver-always-mpg48943.html', 'https://www.jumia.ug/always-stainless-steel-vacuum-flask-silver-5196899.html']
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC