Search code examples
pythonselenium-webdriver

Selecting dynamically generated elements using Selenium

I am wondering how can I get dynamically generated contents using Selenium. Previously mentioned solutions (e.g., 1) did not work in my case. I give an example to make the problem clear. Specifically, I cannot select the dynamically generated elements using Selenium in this page.

Here's a sample python code I am using:

from selenium import webdriver
from selenium.webdriver.common.by import By

if __name__ == '__main__':
    driver = webdriver.Chrome()
    driver.get("https://www.boplatssyd.se/mypages/app")
    try:
        html = driver.find_element(by=By.TAG_NAME, value='html').get_attribute('innerHTML')
        print(html)

    except Exception as e:
        print("An error occurred:", e)

The result is similar to when I get the page source from chrome but if you notice, dynamic elements are not present:

<!DOCTYPE html>
<html lang="sv">
<head>
  
    <meta charset="utf-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1" />

    <link rel="icon" href="/mypages/app/favicon.ico" />
    <link rel="apple-touch-icon" sizes="180x180" href="/mypages/app/apple-touch-icon.png" />
    <link rel="icon" type="image/png" sizes="32x32" href="/mypages/app/favicon-32x32.png" />
    <link rel="icon" type="image/png" sizes="16x16" href="/mypages/app/favicon-16x16.png" />
    <link rel="manifest" href="/mypages/app/manifest.json" />
    <link rel="mask-icon" href="/mypages/app/safari-pinned-tab.svg" color="#5bbad5" />
    <meta name="msapplication-TileColor" content="#68b7d4" />
    <meta name="theme-color" content="#68b7d4" />

    <title>Mina sidor</title>
    <meta name="description" content="Sök bland lediga bostäder i hela regionen." />

    <meta property="og:title" content="Lediga bostäder" />
    <meta property="og:description" content="Sök bland lediga bostäder i hela regionen." />
    <meta property="og:url" content="https://www.boplatssyd.se/mypages/app/" />
    <meta property="og:image" content="" />

    <meta property="og:image:width" content="1200" />
    <meta property="og:image:height" content="630" />
    <meta property="og:type" content="website" />
    <meta property="og:locale" content="sv_SE" />
    <script type="module" crossorigin src="/mypages/app/assets/index-ca5ddbcc.js"></script>
    <link rel="modulepreload" crossorigin href="/mypages/app/assets/@util-e6166646.js">
    <link rel="modulepreload" crossorigin href="/mypages/app/assets/@bootstrap-17c847c2.js">
    <link rel="modulepreload" crossorigin href="/mypages/app/assets/vendor-a4a66ed9.js">
    <link rel="stylesheet" href="/mypages/app/assets/index-20b264da.css">
  
</head>
<body>
  
    <noscript>You need to enable JavaScript to run this app.</noscript>
    <div id="root"></div>
    
  
</body>
</html>

Specifically, the dynamic content should be generated in the following div:

<div id="root"></div>

I can only see the dynamic content if I press Inspect Element on a dynamically generated element. I appreciate your insight on this matter.

Solution

  • You're not waiting for the page to fully load, here's how to wait for a specific element, for example footer:

    from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

driver = webdriver.Chrome()
driver.get('https://www.boplatssyd.se/mypages/app/')

footer = WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.TAG_NAME, 'footer'))) 
html = driver.page_source
print(html)