I need to get the latitude and longitude locations of a lot of addresses, roughly 100k+. However, the way my code works, I think Bing is blocking me as they think I'm a spam bot or something? I'm not exactly sure what the error means. Is there any way to work around this?
for i in listAddresses:
url = "http://dev.virtualearth.net/REST/v1/Locations/US/" + i + "?o=xml&key=" + subscriptionKey
# original code but I got the HTTPError
# page = urllib.request.urlopen(url)
# stringPage = page.read().decode('utf-8')
# some code I found online. Still returns the same error
req = Request(url, headers={'User-Agent': 'Mozilla/5.0'})
stringPage = urlopen(req).read().decode('utf-8')
# really bad/ basic way of getting the latitude and longitude from the xml return
lat = re.findall(r'<Latitude>(.*?)</Latitude>', stringPage)[0]
long = re.findall(r'<Longitude>(.*?)</Longitude>', stringPage)[0]
address = i.replace("%20", " ")
location.append(address)
location.append(lat)
location.append(long)
File "C:\Users\...\lib\urllib\request.py", line 649, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
It is Bing 'blocking' you. It doesn't necessarily believe you are a 'spam bot' - it is simply an imposed limitation (but is used to prevent spam as well) of the free "Basic" account for the Bing Maps REST API.
Bing allows for up to 5 requests a second and a total of 125,000 requests a year for the free account, and paid offerings can have up to 50 requests a second and many more requests. You can view more information here: https://www.microsoft.com/en-us/maps/licensing/options
As you mentioned in the comment, it blocks "after 5 requests" which makes sense. I assume you are using a free account and the Python program is running fast enough in order to make >5 requests a second. Unfortunately, there is no such thing as a free lunch. You'll have to switch providers or pay up.
Alternatively, import time and add a time.sleep(number_of_seconds) in the loop.