Search code examples
pythonseleniumwebdrivergeckodriver

Trying to scrape data from a hover popup with Selenium and Python

I can't seem to select the correct element to trigger Selenium's hover actions. I'm able to use cookies to login, and can see the page load, then scroll down, but I must be selecting the incorrect xpath element, and cannot complete the hover to grab data from the desired table. W/o a login, you won't be able to see the table I see, so I'll place some of the HTML below, along with the error message I get, plus a screenshot:

scrape hover popup with Firefox selenium

from selenium import webdriver
from selenium.webdriver.firefox import firefox_profile
from selenium.webdriver.common.action_chains import ActionChains

from selenium.webdriver.common.keys import Keys
import numpy as np
import pandas as pd
import re

url = 'https://www.oddsportal.com/tennis/czech-republic/wta-ostrava/dodin-oceane-linette-magda-h4MPchI4/'

fp = webdriver.FirefoxProfile('/Users/Frontwing/Library/Application Support/Firefox/Profiles/zku6ulmv.ProxyTest')
driver = webdriver.Firefox(executable_path="config/geckodriver",firefox_profile=fp)
driver.get(url)

items = driver.find_element_by_xpath("/html/body/div[1]/div/div[2]/div[6]/div[1]/div/div[1]/div[2]/div[1]/div[7]/div[3]") \
.get_attribute("innerHTML") 



def get_cash_data():
    # I. Get the raw data by hovering and collecting
    xpath = "/html/body/div[1]/div/div[2]/div[6]/div[1]/div/div[1]/div[2]/div[1]/div[7]/div[3]"
    
    data = driver.find_element_by_xpath(xpath)
    driver.execute_script("arguments[0].scrollIntoView();", data)
    hov = ActionChains(driver).move_to_element(data)
    

    hov.perform()
    data_in_the_bubble = driver.find_element_by_xpath(xpath)
    hover_data = data_in_the_bubble.get_attribute("innerHTML")

    # II. Extract opening odds
    b = re.split('<br>', hover_data)
    #c = re.split('\(([^()]\d+)\)', b)
    #opening_odd = c

    #print(opening_odd)
    print(hover_data)
    return(b)

#print([x.text for x in items])
get_cash_data()

Here is the HTML snippet of the target I want to hover over:

<td class="right odds up"><div onmouseout="delayHideTip()" onmouseover="page.hist(this,'E-0.00-0-0','55fmkx2s5a4x0xdlo2u',44,event,0,1)" style="display: block;">2.42<br>(42)</div></td>

I figured xpath might be an easier way to pinpoint the element, but perhaps not? Here's the error message:

File "opstest.py", line 44, in get_cash_data() selenium.common.exceptions.NoSuchElementException: Message: Unable to locate element: .right odds

Particularly, I suspect the issue has to do with the xpath and likely also the hov workflow. I must not be doing it correctly, or in the correct order..

Solution

  • I did not login here, but I can see all the tr and td's, may they are different but scraping the tooltip logic will remain same.

    1. Launch the browser in full screen mode.
    2. Use Explicit waits for more stability.
    3. Prefer css over xpath.
    4. Always use Relative xpath not absolute xpath.

    Sample code:

    driver = webdriver.Chrome(driver_path)
driver.maximize_window()
#driver.implicitly_wait(30)
wait = WebDriverWait(driver, 50)
action = ActionChains(driver)
driver.get("https://www.oddsportal.com/tennis/czech-republic/wta-ostrava/dodin-oceane-linette-magda-h4MPchI4/")
first_td = WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//tr[@class='lo odd']/td[2]")))
ActionChains(driver).move_to_element(first_td).perform()
tool_tip_text_container = driver.find_element(By.CSS_SELECTOR, "span#tooltiptext").get_attribute('innerText')
print(tool_tip_text_container)

    Imports :

    from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.action_chains import ActionChains

    Output :

    20 Sep, 17:42 2.30 +0.03
20 Sep, 17:29 2.27 +0.02
20 Sep, 17:25 2.25 -0.20
20 Sep, 17:24 2.45 +0.20
20 Sep, 17:20 2.25 -0.20
20 Sep, 17:19 2.45 +0.20
20 Sep, 16:58 2.25 -0.20
20 Sep, 16:56 2.45 +0.20
20 Sep, 16:30 2.25 -0.20
20 Sep, 16:29 2.45 +0.18
20 Sep, 16:23 2.27 -0.18
20 Sep, 16:21 2.45 +0.20
20 Sep, 15:52 2.25 -0.20
20 Sep, 15:51 2.45 +0.20
20 Sep, 15:45 2.25 -0.20
20 Sep, 15:42 2.45 +0.20
20 Sep, 15:41 2.25 -0.20
20 Sep, 15:36 2.45 +0.12
20 Sep, 15:16 2.33 -0.12
20 Sep, 15:14 2.45 +0.12

Opening odds:
19 Sep, 19:00 2.33