How to wait for text value to show using Selenium Python

Question

I am trying to get the value of an element that renders text upon clicking a dropdown. I am currently using implicity_wait() to make sure the element is appearing, but when I run the script, the .text call returns empty strings. If I slowly run each line of the script the .text values populate. Based on this i assume that I have to wait for the text to render, but I can't work out how to do this.

Looking at the expected conditions documentation all the of the text_to_be_present_... conditions want me to know what text I am waiting for. Since I am webscraping I don't know this and so I am trying to pass a regex condition to the text_ argument, that matches a generic form of the value I am looking for. I am not getting the expected result with the value still returning an empty string when I run the script.

Here is the code I am trying:

from selenium import webdriver 
from selenium.webdriver.chrome.options import Options 
from selenium.webdriver.support import expected_conditions as EC 
from selenium.webdriver.support.ui import WebDriverWait

#Set the options for running selenium as headless
options = Options()
options.headless = True
options.add_argument("--window-size=1920,1200")

#Create the driver object
driver = webdriver.Chrome(options=options, executable_path=DRIVER_PATH)
driver.implicitly_wait(10)

output = []
driver.get(html)
nat_res_element = driver.find_element_by_xpath('//*[@id="accordion-theme"]/div[1]/div[1]/span')
nat_res_element.click()
element = WebDriverWait(driver, 10).until(EC.text_to_be_present_in_element_value(locator = By.xpath('//*[@id="collapse0"]/div/div/ul/li/span[2]'), text_ = '[\d].*'))
output.append(element.text)

The url is: https://projects.worldbank.org/en/projects-operations/project-detail/P159382. I am trying to access the values under the 'Environment and Natural Resource Management' dropdown. Since this is digit; digit; %, I am trying regex [\d].*.

Welcome a way to handle this.

Answer 1

climate_change = WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.XPATH, '(//*[@class="twolevel"]//li//span)[2]'))).text
adaptation = WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.XPATH, '(//*[@class="twolevel"]//li//span)[4]'))).text
mitigation = WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.XPATH, '(//*[@class="twolevel"]//li//span)[6]'))).text

The above xpath expressions will pull the desired data from the 'Environment and Natural Resource Management' dropdown.

It's working fine with non-headless browser.

Full Script:

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

options = Options()
options.add_argument("--window-size=1920,1200")
#options.add_argument("--headless")


s = Service("./chromedriver") ## path to where you saved chromedriver binary
driver = webdriver.Chrome(service=s, options=options)

url = 'https://projects.worldbank.org/en/projects-operations/project-detail/P159382'
driver.get(url)
time.sleep(5)

nat_res_element = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, '//*[@id="accordion-theme"]/div[1]/div[1]/span')))
nat_res_element.click()
data=[]
climate_change = WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.XPATH, '(//*[@class="twolevel"]//li//span)[2]'))).text
adaptation = WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.XPATH, '(//*[@class="twolevel"]//li//span)[4]'))).text
mitigation = WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.XPATH, '(//*[@class="twolevel"]//li//span)[6]'))).text
data.append({
    'Climate change':climate_change,
    'Adaptation':adaptation,
    'Mitigation':mitigation
    })

print(data)

driver.quit()
  

Output:

[{'Climate change': '64%', 'Adaptation': '32%', 'Mitigation': '32%'}]