Search code examples
pythonhttpx

Why the response of httpx is different compared to inspect element network?


I am trying to scrape this site https://shopee.co.id/S-B-Golden-Curry-Bumbu-Kari-Japanese-Curry-Mix-Medium-Hot-220-gr-i.65323877.3343602079?sp_atk=0b471505-cfca-491b-b7a2-453d455eecf6&xptdk=0b471505-cfca-491b-b7a2-453d455eecf6

I am using python httpx and the below is my code

url=f"https://shopee.co.id/api/v4/item/get?shopid=65323877&itemid=3343602079"
headers = { "User-Agent" : "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36"
            }
resp=httpx.get(url,headers=headers)

The returned json is as follows:

{'error': None,
 'error_msg': None,
 'data': {'itemid': 3343602079,
  'shopid': 65323877,
  'userid': 0,
  'price_max_before_discount': -1,
  'has_lowest_price_guarantee': False,
  'price_before_discount': 0,
  'price_min_before_discount': -1,
  'exclusive_price_info': None,
  'hidden_price_display': None,
  'price_min': 9315023372,
  'price_max': 9315023372,
  'price': 9315023372, ...

I see that the price shown in the json is different than the one on the web;7490000000. Can someone please help?


Solution

  • Try to send a request without User-Agent header. It seems that the server sends different content based on this information:

    import requests
    
    url = 'https://shopee.co.id/api/v4/item/get?shopid=65323877&itemid=3343602079'
    
    data = requests.get(url).json()
    print(data)
    

    Prints:

    {
        "error": None,
        "error_msg": None,
        "data": {
            "itemid": 3343602079,
            "shopid": 65323877,
            "userid": 0,
            "price_max_before_discount": -1,
            "has_lowest_price_guarantee": False,
            "price_before_discount": 0,
            "price_min_before_discount": -1,
            "exclusive_price_info": None,
            "hidden_price_display": None,
            "price_min": 7490000000,
            "price_max": 7490000000,
            "price": 7490000000,
            "stock": 0,
    
    ...