How to get past cookie popup with selenium in python

Question

I am trying to scrape data from a property website but cannot get past the cookie popup. I am using Selenium and Python 3 and have searched the web for possibl solutions. Tried everything I could find, but nothing seems to work. I'm sure it's really easy, if you know what you are doing. :-)

Can anyone push me in the right direction please? I am fairly new to Python so please keep it simple.

I have tried a few things with my code:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import NoSuchElementException
from selenium.common.exceptions import TimeoutException
import time

HEMNET_URL = "https://www.hemnet.se/"
chrome_options = webdriver.ChromeOptions()
chrome_options.add_experimental_option("detach", True)
driver = webdriver.Chrome(options=chrome_options)
driver.get(HEMNET_URL)

time.sleep(5)


def cookie_click_1():
    try:
        cookie_button = driver.find_element(By.CSS_SELECTOR, value="Acceptera alla")
        cookie_button.click()
    except NoSuchElementException:
        print("no cookie popup")


def cookie_click_2():
    try:
        cookie_button = WebDriverWait(driver, 5).until(EC.element_to_be_clickable((By.CLASS_NAME, 'sc-dcJsrY kIAuBW')))
        cookie_button.click()
    except TimeoutException:
        print("no cookie popup")


def cookie_click_3():
    try:
        cookie_button = driver.find_element(By.XPATH, value="//button[normalize-space()='Acceptera alla']")
        cookie_button.click()
    except NoSuchElementException:
        print("no cookie popup")


def cookie_click_4():
    try:
        cookie_button = driver.find_element(By.LINK_TEXT, value="Acceptera alla")
        cookie_button.click()
    except NoSuchElementException:
        print("no cookie popup")


cookie_click_1()
cookie_click_2()
cookie_click_3()
cookie_click_4()

This is the button I am trying for Selenium to click ("Accepta alla"):

Original website code

There is no obvious class (is it dynamic?) I can use with 'find.element', and no name or id.

Running any of the functions above results in "no cookie popup".

What I am doing wrong please?

I tried all suggestions I found online, but for some reason none of them worked. What I expect to happen is that Selenium clicks on the "Acceptera alla" button so I can get to the login page.

I am missing something here, or is there something special about this website's code?

Answer 1

That element is enclosed in a shadow-root parent element. Here is one way of accepting cookies:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import Select
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

chrome_options = Options()
chrome_options.add_argument("--no-sandbox")
chrome_options.add_argument('disable-notifications')
chrome_options.add_argument("window-size=1280,1080")

driver = webdriver.Chrome(options=chrome_options)
wait = WebDriverWait(driver, 15)
driver.get('https://www.hemnet.se/')
print('visited page')
parent_el_shadow_root = wait.until(EC.presence_of_element_located((By.XPATH, '//div[@id="usercentrics-root"]'))).shadow_root
WebDriverWait(parent_el_shadow_root, 15).until(EC.presence_of_element_located((By.CSS_SELECTOR, 'button[data-testid="uc-accept-all-button"]'))).click()
print('accepted cookies') 

See Selenium documentation here.