I'm trying to run the following snippet of code to retrieve data from the specified url. I tried using the 'timeout=5' parameter as well as leaving it out.
The end result is that running the script either hangs python or I get a timeout error message? Opening the url in a browser seems to return valid json, but I can't seem to scrape the data in python.
What's the deal?
import requests
url = "http://stats.nba.com/stats/shotchartdetail?Period=0&VsConference=&LeagueID=00&LastNGames=0&TeamID=0&Position=&Location=&Outcome=&ContextMeasure=FGA&DateFrom=&StartPeriod=&DateTo=&OpponentTeamID=0&ContextFilter=&RangeType=&Season=2016-17&AheadBehind=&PlayerID=202738&EndRange=&VsDivision=&PointDiff=&RookieYear=&GameSegment=&Month=0&ClutchTime=&StartRange=&EndPeriod=&SeasonType=Regular+Season&SeasonSegment=&GameID=&PlayerPosition="
response = requests.get(url,timeout=5)
print(response)
you are not passing headers and specifically user-agent in headers
The User-Agent request header contains a characteristic string that allows the network protocol peers to identify the application type, operating system, software vendor or software version of the requesting software user agent.
headers ={"User-Agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/60.0.3112.78 Chrome/60.0.3112.78 Safari/537.36"}
response = requests.get(url,headers=headers,timeout=5)
print response.text