Selenium + BS4 retrieve hidden element

Question

I wrote the following code to retrieve a div with the class "tab-statistics-1-statistic". This one is nested in the "statistics-content" div.

soup.find(id="statistics-content").find(id="tab-statistics-1-statistic")

But when I print the output of the above line it only returns "div id='statistics-content'>" even though in "Inspect element" the div contains nested elements. The page used for this code is https://www.flashscore.com/match/CM2dBaSF/#match-summary

How can I get access to the nested elements within the div ?

Answer 1

You are navigating to landing page but need to click through to the statistics tab to generate the required html. You need to allow time for javascript to run to populate info.

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait 
from selenium.webdriver.support import expected_conditions as EC
d = webdriver.Chrome()
url ='https://www.flashscore.com/match/CM2dBaSF/#match-summary'

d.get(url)
d.find_element_by_id('a-match-statistics').click()
item = WebDriverWait(d, 5).until(EC.presence_of_element_located((By.ID , 'tab-statistics-0-statistic')))
print(item.text)
d.quit()

---

As a new url is generated for the stats tab you can just use that direct instead

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait 
from selenium.webdriver.support import expected_conditions as EC
d = webdriver.Chrome()
urlDirect = 'https://www.flashscore.com/match/CM2dBaSF/#match-statistics;0'
d.get(urlDirect)
item = WebDriverWait(d, 5).until(EC.presence_of_element_located((By.ID , 'tab-statistics-0-statistic')))
print(item.text)
d.quit()