I am wondering how can I get dynamically generated contents using Selenium. Previously mentioned solutions (e.g., 1) did not work in my case. I give an example to make the problem clear. Specifically, I cannot select the dynamically generated elements using Selenium in this page.
Here's a sample python code I am using:
from selenium import webdriver
from selenium.webdriver.common.by import By
if __name__ == '__main__':
driver = webdriver.Chrome()
driver.get("https://www.boplatssyd.se/mypages/app")
try:
html = driver.find_element(by=By.TAG_NAME, value='html').get_attribute('innerHTML')
print(html)
except Exception as e:
print("An error occurred:", e)
The result is similar to when I get the page source from chrome but if you notice, dynamic elements are not present:
<!DOCTYPE html>
<html lang="sv">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<link rel="icon" href="/mypages/app/favicon.ico" />
<link rel="apple-touch-icon" sizes="180x180" href="/mypages/app/apple-touch-icon.png" />
<link rel="icon" type="image/png" sizes="32x32" href="/mypages/app/favicon-32x32.png" />
<link rel="icon" type="image/png" sizes="16x16" href="/mypages/app/favicon-16x16.png" />
<link rel="manifest" href="/mypages/app/manifest.json" />
<link rel="mask-icon" href="/mypages/app/safari-pinned-tab.svg" color="#5bbad5" />
<meta name="msapplication-TileColor" content="#68b7d4" />
<meta name="theme-color" content="#68b7d4" />
<title>Mina sidor</title>
<meta name="description" content="Sök bland lediga bostäder i hela regionen." />
<meta property="og:title" content="Lediga bostäder" />
<meta property="og:description" content="Sök bland lediga bostäder i hela regionen." />
<meta property="og:url" content="https://www.boplatssyd.se/mypages/app/" />
<meta property="og:image" content="" />
<meta property="og:image:width" content="1200" />
<meta property="og:image:height" content="630" />
<meta property="og:type" content="website" />
<meta property="og:locale" content="sv_SE" />
<script type="module" crossorigin src="/mypages/app/assets/index-ca5ddbcc.js"></script>
<link rel="modulepreload" crossorigin href="/mypages/app/assets/@util-e6166646.js">
<link rel="modulepreload" crossorigin href="/mypages/app/assets/@bootstrap-17c847c2.js">
<link rel="modulepreload" crossorigin href="/mypages/app/assets/vendor-a4a66ed9.js">
<link rel="stylesheet" href="/mypages/app/assets/index-20b264da.css">
</head>
<body>
<noscript>You need to enable JavaScript to run this app.</noscript>
<div id="root"></div>
</body>
</html>
Specifically, the dynamic content should be generated in the following div:
<div id="root"></div>
I can only see the dynamic content if I press Inspect Element on a dynamically generated element. I appreciate your insight on this matter.
You're not waiting for the page to fully load, here's how to wait for a specific element, for example footer
:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Chrome()
driver.get('https://www.boplatssyd.se/mypages/app/')
footer = WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.TAG_NAME, 'footer')))
html = driver.page_source
print(html)