I have attached a screenshot of the HTML of the website I am trying to scrape, there is a table and within it I want to get some of the data from the rows which are within body (they clearly exist), however, it was not working so I decided to print tbody which revealed that the parsing found the table and tbody but not the rows within. I don't know how to fix this any help would be appreciated.HTML from wesbite
This is the beginning of my code:
url = "https://superfpl.com/player_stats"
results = requests.get(url)
soup = BeautifulSoup(results.text, "html.parser")
players = []
teams = []
positions = []
ownerships = []
print(soup.find("tbody"))
player_div = soup.find_all('tr', role_="row", class_="odd")
The data is loaded dynamically through JavaScript. But you can simulate it with the requests
module:
import requests
url = 'https://superfpl.com/ajax/player_stats'
data = requests.get(url).json()
# uncomment this to see all data (WARNING, huge list!):
# import json
# print(json.dumps(data, indent=4))
# pretty print some data to screen:
for row in data['data']:
print('{:<20}{:<20}{:<20}'.format(row['web_name'], row['position'], row['points_per_game']))
Prints:
Connolly FWD 2.4
Cresswell DEF 2.7
Lennon MID 0.9
Mooy MID 2.9
Ramsdale GKP 3.4
Wan-Bissaka DEF 3.3
Koiki DEF 0.0
Doucouré MID 3.3
Idah FWD 1.3
Lallana MID 1.8
Masina DEF 2.8
Adam Smith DEF 2.2
... and so on.