I am scraping using postman and I have located what data I need I just need it to work locally. Although it works in postman and in kaggle but not in local machine neither in colab. Here's the code for reference: `
import requests
url = "https://apisae1.phdvasia.com/v1/product-hut-fe/localizations?order_type=C&disposition=C&limit=4"
payload = {}
headers = {
'authority': 'apisae1.phdvasia.com',
'accept': 'application/json, text/plain, */*',
'accept-language': 'en-US,en;q=0.9',
'client': '57950680-0208-4703-9e0b-c5a8cc45cf42',
'lang': 'pt',
'origin': 'https://www.pizzahut.com.br',
'referer': 'https://www.pizzahut.com.br/',
'sec-ch-ua': '"Not.A/Brand";v="8", "Chromium";v="114", "Google Chrome";v="114"',
'sec-ch-ua-mobile': '?0',
'sec-ch-ua-platform': '"Windows"',
'sec-fetch-dest': 'empty',
'sec-fetch-mode': 'cors',
'sec-fetch-site': 'cross-site',
'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36 OPR/99.0.0.0',
'Cookie': 'ak_bmsc=CA52AD92D65BE36761FAC7C181F253D0~000000000000000000000000000000~YAAQxGvcF7RDj0SJAQAALPEpSRSGE7kXTAwqrRr6W7FKHAmAiIZLed4usMa8hURw56blJZSho5ANdgoKbhEiJCGu8lFXst2L2uWHVaoNXi3QZgkcGHgYmHvWNw94pK9aoh4zr9YvHaeMFRd5Xr01mSDpSfjtx7PwGydeGkLcRo+2PX6xlXM9UPSDal59DPOjkM0LclJgl0s3FOumelzJrL5f73VPNQBEmh9Ek+XInsBiO5z6x118opcFaDNoRGAW/8rnaYJgQNF/t7WgfddKr7VA6Dif7s6mA0kxiUYlXpjQLuaFvCNc++RO6+fqm0fPz8bz8BQPKg9OIjzDPSdXUrRMkpQhdmImyzNDv/v2MePK8H2BYZnjh1JK0Kuw; bm_sv=FBC11E925F19B3C0A6BA6708D42A3D3A~YAAQxGvcF8yxkESJAQAAFkxASRT933k/B3sEomiSPBAzN4AEskTpDt7ilDa1EZwRgSq0vPW3eeF1pcmF8mfeEzeUTd9M9cLkVh3FoENhHyaorUh9yPYxu06HHCdXltInBWkY5jXsNa18804DF6tVDieU0ZZd4cKm8o1AMB1luUPGsLPJewgPVQWNWMeKAz6PyZVo+qAcZPW67ACmEzr4J+SONmXPAiPlK0jn77bU8PEOWAukDLbeO6BzwGFm6a4ESjr3~1'
}
response = requests.request("GET", url, headers=headers, data=payload, allow_redirects=False)
print(response.text)
I tried changing user agent to random and even current version of browser I'm using and tried on other cloud services. It worked on kaggle but not on colaband local machine. PLs help
The only header value you need is the client.
import requests
import json # only used for presentation
url = "https://apisae1.phdvasia.com/v1/product-hut-fe/localizations?order_type=C&disposition=C&limit=4"
headers = {
'client': '57950680-0208-4703-9e0b-c5a8cc45cf42'
}
with requests.get(url, headers=headers) as response:
response.raise_for_status()
print(json.dumps(response.json(), indent=2))
Without the client header (or an invalid client) you'll get HTTP 400