Web Element (button) not found by Selenium (Python)

Question

I am trying to set cookie preferences (two button clicks) or even just accept cookies (one click).

When the page I am trying to interact with renders it has an overlay over the DOM preventing interaction until cookie prefs are set. I have checked the hierarchy as best I can and I am sure the cookie buttons are not in an iframe (so I don't need to switch to iframe click and then switch back).

My code gets all elements that match - tried tag name, and css selector. I get matches in a list, I then iterate them and check if the button has matching text. My code to set cookie prefs and save gets three buttons in the DOM which are non interactable (because of the overlay div) but not the cookie buttons. The code (below) to find the Accept cookies button does not find any matches.

URL = https://glaplanningapps.commonplace.is/planningapps/PA-21-02377

Here is how I try to get the "Accept Cookies" button:

elements = driver.find_elements('css selector', 'button.sc-dcJsrY.CjGAy')
for element in elements:
    try:
        element_text = element.text
    except:
        element_text = ''
    if element_text == 'Accept all cookies':
        return True

if I switch out the first line to this:

elements = driver.find_elements('tag name', 'button')

I still get no matches :(

Example html: Accept cookies:

<button role="button" data-testid="uc-accept-all-button" style="margin: 0px 6px;" class="sc-dcJsrY CjGAy">Accept all cookies</button>

Python 3.10 Selenium 4.16.0 Firefox, geckodriver 0.33.0

Answer 1

Your button is placed inside shadow-root, to get internal shadow root structure, you should get it's host first and then get shadowRoot property.

Shadow host in current example is element with locator #usercentrics-root

get_shadow_root function gets shadow root from host, using JS executor.

More about Shadow DOM

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

driver = webdriver.Chrome()
timeout = 10
wait = WebDriverWait(driver, timeout)

def get_shadow_root(element):
    return driver.execute_script('return arguments[0].shadowRoot', element)


driver.get("https://glaplanningapps.commonplace.is/planningapps/PA-21-02377")
driver.maximize_window()

shadow_host = wait.until(EC.presence_of_element_located((By.ID, 'usercentrics-root')))
shadow_container = get_shadow_root(shadow_host).find_element(By.CSS_SELECTOR, '[data-testid=uc-app-container]')
WebDriverWait(shadow_container, timeout).until(EC.presence_of_element_located((By.CSS_SELECTOR, '[data-testid=uc-accept-all-button]'))).click()
wait.until(EC.invisibility_of_element_located((By.ID, 'usercentrics-root')))