Search code examples
pythonhtmlweb-scrapingbeautifulsoupurllib

Why div returns empty when web-scraping Steam Game List?

I'm new in web-scraping and using BeautifulSoup4, so I'm sorry if my question is obvious.

I'm trying to get the hours played from Steam, but <div id="games_list_rows" style="position: relative"> returns None when it should return a lot of differents <div class="gameListRow" id="game_730"> with stuff inside.

I've tried with a friend's profile who has a few games because I was thinking that working with a lot of data could make BS4 ignore the div, but it keeps showing the div empty.

Here's my code:

import bs4 as bs
import urllib.request

# Retrieve profile
profile = "chubaquin"#input("enter profile: >")
search = "https://steamcommunity.com/id/"+profile+"/games/?tab=all"

sauce = urllib.request.urlopen(search)
soup = bs.BeautifulSoup(sauce, "lxml")

a = soup.find("div", id="games_list_rows")

print(a)

Thanks for your help!
Solution

  • The website is loaded dynamically, therefore requests doesn't support it. Try using Selenium as an alternative to scrape the page.

    Install it with: pip install selenium.

    Download the correct ChromeDriver from here.

    from time import sleep
from bs4 import BeautifulSoup
from selenium import webdriver


url = "https://steamcommunity.com/id/chubaquin/games/?tab=all"
driver = webdriver.Chrome(r"c:\path\to\chromedriver.exe")
driver.get(url)
# Wait for the page to fully render before parsing it
sleep(5)
soup = BeautifulSoup(driver.page_source, "html.parser")

print(soup.find("div", id="games_list_rows"))