Search code examples
pythonweb-scrapingbeautifulsouphtml-tablehtml-parsing

Beautiful Soup not parsing all tags


I have attached a screenshot of the HTML of the website I am trying to scrape, there is a table and within it I want to get some of the data from the rows which are within body (they clearly exist), however, it was not working so I decided to print tbody which revealed that the parsing found the table and tbody but not the rows within. I don't know how to fix this any help would be appreciated.HTML from wesbite

Output when printing body

This is the beginning of my code:

url = "https://superfpl.com/player_stats"

results = requests.get(url)

soup = BeautifulSoup(results.text, "html.parser")

players = []

teams = []

positions = []

ownerships = []

print(soup.find("tbody"))

player_div = soup.find_all('tr', role_="row",  class_="odd")

Solution

  • The data is loaded dynamically through JavaScript. But you can simulate it with the requests module:

    import requests
    
    url = 'https://superfpl.com/ajax/player_stats'
    data = requests.get(url).json()
    
    # uncomment this to see all data (WARNING, huge list!):
    # import json
    # print(json.dumps(data, indent=4))
    
    # pretty print some data to screen:
    for row in data['data']:
        print('{:<20}{:<20}{:<20}'.format(row['web_name'], row['position'], row['points_per_game']))
    

    Prints:

    Connolly            FWD                 2.4                 
    Cresswell           DEF                 2.7                 
    Lennon              MID                 0.9                 
    Mooy                MID                 2.9                 
    Ramsdale            GKP                 3.4                 
    Wan-Bissaka         DEF                 3.3                 
    Koiki               DEF                 0.0                 
    Doucouré            MID                 3.3                 
    Idah                FWD                 1.3                 
    Lallana             MID                 1.8                 
    Masina              DEF                 2.8                 
    Adam Smith          DEF                 2.2                 
    
    ... and so on.