Selenium: How to fetch elements inside #shadow-root (open) that includes a comment

Question

I am trying to fetch an element from html page which is hidden deep in multiple shadow roots with selenium

shadow_element = driver.execute_script("return document.querySelector('w3m-modal').shadowRoot.querySelector('div').querySelector('div')")
print(shadow_element.get_attribute('innerHTML'))

Code above reaches a div on the path to my target element and prints out inner html of this div. First line of its inner html is:

<!--?lit$507756472$-->

Which is exactly what my code prints out. But after this line there are 2 more html elements which are ignored by selenium. How do I reach elements below the comment?

HTML:

Below is html of w3m-modal (this is the first element I am selecting with querySelector). What I need to get is <div class="w3m-card">

<w3m-modal>
#shadow-root (open)
<!---->
<w3m-explorer-context></w3m-explorer-context>
<w3m-theme-context></w3m-theme-context>
<w3m-wc-connection-context>
</w3m-wc-connection-context>
<w3m-account-context>
</w3m-account-context>
<w3m-network-context>
</w3m-network-context>
<div id="w3m-modal" role="alertdialog" aria-modal="true" class="w3m-overlay w3m-active" style="opacity: 1;">
    <div class="w3m-container" tabindex="0" style="transform: scale(var(--motion-scale)); --motion-scale: 1;">
        <!--?lit$656149617$-->
        <w3m-modal-backcard></w3m-modal-backcard>
        <div class="w3m-card">
            <w3m-modal-router></w3m-modal-router>
            <w3m-modal-toast></w3m-modal-toast>
        </div>
    </div>
</div>
</w3m-modal>

Answer 1

To extract the HTML within the #shadow-root (open) you need to induce WebDriverWait for the visibility_of(element) and you can use the following locator strategy:

print(WebDriverWait(driver, 20).until(EC.visibility_of((driver.execute_script("return document.querySelector('w3m-modal').shadowRoot.querySelector('div.w3m-container')")))).get_attribute('innerHTML'))

Note : You have to add the following imports :

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC