Search code examples
pythonxpathhref

Scrape URL links with Python

Here is my code:

from selenium import webdriver
from bs4 import BeautifulSoup

driver = webdriver.Firefox()
url = 'https://www.coteur.com/cotes-foot.php'
driver.get(url)

links = driver.find_elements_by_xpath('//a[contains(@href, "match/cotes-")]')

driver.close()

I want to scrape all the URL links associated with the soccer games from this site: https://www.coteur.com/cotes-foot.php

I always scrape all the <a> elements where the soccer games are included. But how can I extract the URLs linked to these soccer games?

Solution

  • you are getting webelements with find_elements_by_xpath you need to get href from it

    from selenium import webdriver

driver = webdriver.Firefox()
url = 'https://www.coteur.com/cotes-foot.php'
driver.get(url)

links = []
for i in driver.find_elements_by_xpath('//a[contains(@href, "match/cotes-")]'):
    links.append(i.get_attribute('href'))

print(links)
driver.close()