400 Bad Request for legitimate url when using Python's Request library

Question

I'm trying to use Python's Requests library to obtain a JSON object from a call to requests.get():

import requests

url = 'http://stats.nba.com/stats/boxscoretraditionalv2'+\
'?EndPeriod=10&EndRange=28800&GameID=0021500893&RangeType=0'+\
'&Season=2015-16&SeasonType=Regular+Season&StartPeriod=1'+\
'&StartRange=0'

# is the auth necessary?
response = requests.get(url, auth=('user', 'pass'))

And I get the following exception:

Traceback (most recent call last):
  File "sample.py", line 22, in <module>
    response.raise_for_status() # raise exception if invalid response
  File "/Library/Python/2.7/site-packages/requests/models.py", line 840, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: http://stats.nba.com/stats/boxscoretraditionalv2?EndPeriod=10&EndRange=28800&GameID=0021500893&RangeType=0&Season=2015-16&SeasonType=Regular+Season&StartPeriod=1&StartRange=0

What do I need to do in order to fix this? The link works - it simply renders the JSON, and returns a 200 OK if viewed through Chrome.

Answer 1

Add headers to your request:

import requests

url = 'http://stats.nba.com/stats/boxscoretraditionalv2'+\
'?EndPeriod=10&EndRange=28800&GameID=0021500893&RangeType=0'+\
'&Season=2015-16&SeasonType=Regular+Season&StartPeriod=1'+\
'&StartRange=0'

headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:39.0) Gecko/20100101 Firefox/39.0'}

response = requests.get(url, headers=headers)

The admin of NBA.com decided to give out HTTP 400s to users without a User-Agent that looks like a normal browser (probably to reduce scraping).