Selenium can't detect text in html from Google Maps

Question

I am trying to scrape travel times between two points from Google Maps:

I have used inspect to find the XPath of the travel time in the HTML file:

The following code raises an exception:

from selenium.webdriver.chrome.service import Service
from selenium import webdriver
from selenium.webdriver.common.by import By

url = 'https://www.google.com/maps/dir/35.7290613,+51.238014/35.7504171,51.2244444/@35.7296931,51.2580754,13z/data=!4m8!4m7!1m5!1m1!1s0x0:0x3a1893ebcae30b2e!2m2!1d51.238014!2d35.7290613!1m0'

#initialize web driver
with webdriver.Chrome() as driver:
    #navigate to the url
    driver.get(url)
    #find element by xpath
    myDiv = driver.find_element(By.XPATH, '/html/body/div/div[12]/div[2]/div/div[2]/div/div[3]/button[1]/div/div[1]/div/h1/span[2]/span/span')
    print(myDiv)
    print(myDiv.text)
 

The exception:

NoSuchElementException: no such element: Unable to locate element: {"method":"xpath","selector":"/html/body/div/div[12]/div[2]/div/div[2]/div/div[3]/button[1]/div/div[1]/div/h1/span[2]/span/span"} (Session info: chrome=108.0.5359.125)

Are there any workarounds?

Answer 1

1. First, you need to wait for the page to becopme loaded.
2. Then you need to located the correct elements. The elements you want to scrape can be located by this XPath:

"//div[contains(@aria-labelledby,'section-directions-trip-title')]//span[@jstcache='198']"

I see 3 traveling options on that page, all the 3 trip times can be received as folowing:

trip_times = wait.until(EC.visibility_of_all_elements_located((By.XPATH, "//div[contains(@aria-labelledby,'section-directions-trip-title')]//span[@jstcache='198']")))
for trip_time in trip_times:
    print(trip_time.text)

The complete code is as following and it worked correct:

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

options = Options()
options.add_argument("start-maximized")

webdriver_service = Service('C:\webdrivers\chromedriver.exe')
driver = webdriver.Chrome(options=options, service=webdriver_service)
wait = WebDriverWait(driver, 30)

url = "https://www.google.com/maps/dir/35.7290613,+51.238014/35.7504171,51.2244444/@35.7296931,51.2580754,13z/data=!4m8!4m7!1m5!1m1!1s0x0:0x3a1893ebcae30b2e!2m2!1d51.238014!2d35.7290613!1m0"
driver.get(url)

trip_times = wait.until(EC.visibility_of_all_elements_located((By.XPATH, "//div[contains(@aria-labelledby,'section-directions-trip-title')]//span[@jstcache='198']")))
for trip_time in trip_times:
    print(trip_time.text)