Search code examples
pythonselenium-webdriverweb-scraping

Python + Selenium: web scraping

I am trying to extract some information from a website using Selenium below is the link to the website: http://www.ultimatetennisstatistics.com/playerProfile?playerId=4742 The information I am trying to get is players statistics which is located at a dropdown button 'statistics' which takes you to another page I have inspected the button and got an XPath and CSS but when I run my program it does not open the player's statistics page instead it just open this link below: http://www.ultimatetennisstatistics.com/playerProfile?playerId=4742

and gives me an error:

NoSuchElementException: no such element: 

Unable to locate element: {"method":"css selector","selector":"#playerPills > li.dropdown.active.open > ul > li.active"}
  (Session info: chrome=67.0.3396.99)
  (Driver info: chromedriver=2.41.578737 (49da6702b16031c40d63e5618de03a32ff6c197e),platform=Windows NT 6.3.9600 x86_64)

Below is my code:

from bs4 import BeautifulSoup
from selenium import webdriver

driver = webdriver.Chrome()
driver.get("http://www.ultimatetennisstatistics.com/playerProfile?playerId=4742")
soup = BeautifulSoup(driver.page_source,"lxml")

bm = driver.find_element_by_css_selector('#playerPills > li.dropdown.active.open > ul > li.active')
bm.click()

Can someone show we how I can get the player's statistics page to open with Selenium and extract the information in the table?

Solution

  • If you inspect the html source of the page, you can access the CSS id directly for the button you want to click. Using selenium you can find the button by its id by doing driver.find_element_by_id('statisticsPill') which will allow you to then click on it to show the table.
    After that loads you can then parse through the table to get the data you want.

    Example:

    from selenium import webdriver

driver = webdriver.Chrome()
driver.get("http://www.ultimatetennisstatistics.com/playerProfile?playerId=4742")

try:
    # Fist click on the dropdown
    dropdown = driver.find_element_by_xpath("//a[@id='statisticsPill']/../../..")
    dropdown.click()

    # Then click on the statistics button
    bm = driver.find_element_by_id('statisticsPill')
    bm.click()
except NoSuchElementException as e:
    # Do error handling when cannot find the button

    Edit: You have to first click on the dropdown for the button to become visible and then click on it.