"Request to buy" price Steam

Question

I want to get this price from steam market but if i try to get it this way

name = "P250 | Red Rock (Battle-Scarred)"
html = requests.get("https://steamcommunity.com/market/listings/730/"+name).text
soup = BeautifulSoup(html,"html5lib")

I only get None value. Another hand I can use Selenium but it's very slow for me (nearly 3 sec in one request). How to get this number ?

Answer 1

You are getting None because that price element is dynamically added on the web page by some JavaScript. Try finding parent element of the span element (which contains price) with id market_commodity_buyrequests using bs4. You'll see that parent div element has no elements and no text. While in a browser you'll find two span elements and rest would be text nodes. That's the problem.

Using network tools I saw that a JS on the source HTML makes the following request. It uses a numeric ID called item_name_id to identify the product. If you can find this numeric ID, you can construct the URL (returns JSON response) and get the price from the response using the key named buy_order_summary. This key's value has HTML markup and consists of content that is missing in your original requests.get(url) response.

Here's the URL for pricing and sample code to get and use it.

https://steamcommunity.com/market/itemordershistogram?country=US&language=english¤cy=1&item_nameid=175896332&two_factor=0

import re
import requests
from bs4 import BeautifulSoup


# finds id of the product
def get_id(s):
    id = None
    for script in s.find_all('script'):
        id_regex = re.search('Market_LoadOrderSpread\(([ 0-9]+)\)', script.text)
        if id_regex:
            id = id_regex.groups()[0].strip()
            break
    return id

name_url = "https://steamcommunity.com/market/listings/730/P250%20%7C%20Red%20Rock%20%28Battle-Scarred%29"
html = requests.get(name_url).text
soup = BeautifulSoup(html, 'lxml')
id = get_id(soup)

if id:
    id_url = f"https://steamcommunity.com/market/itemordershistogram?country=US&language=english&currency=1&item_nameid={id}&two_factor=0"
    html = requests.get(id_url).json()
    soup = BeautifulSoup(html['buy_order_summary'], 'lxml')

    print(f"Price is {soup.select_one('span:last-child').text}")
else:
    print("Could not get ID")
    exit()

Output:

Price is $3.77