I am getting the following error for the following code and I am not able to solve the error. The error occures after the loop has created some files correctly.
I am using the following code:
import urllib
import json
symbolslist = open("C:/Users/Asus/Desktop/Python/web scraper/scraping/symbols.txt").read()
symbolslist = [s for s in symbolslist.split("\n") if s]
for symbol in symbolslist:
myfile = open("C:/Users/Asus/Desktop/Python/web scraper/scraping/year_data/" + symbol +".txt" , "w+")
myfile.close()
htmltext = urllib.urlopen("http://www.bloomberg.com/markets/api/bulk-time-series/price/"+symbol+"%3AUS?timeFrame=1_YEAR")
data = json.load(htmltext)
try:
datapoints = data[0]["price"]
except KeyError:
print 'Unknown symbol: {0}'.format(symbol)
continue
myfile = open("C:/Users/Asus/Desktop/Python/web scraper/scraping/year_data/" + symbol +".txt" , "a")
for point in datapoints:
myfile.write(str(symbol +","+ str(point['date']) +","+ str(point['value'])+"\n"))
myfile.close()
A symbol is for example AAPL (for Apple) and this is the link to bloomberg data for Apple: http://www.bloomberg.com/markets/api/bulk-time-series/price/AAPL%3AUS?timeFrame=1_YEAR
This is the data:
[{u'lastPrice': 10.18, u'price': [{u'date': u'2014-12-26', u'value': 10.01}, {u'date': u'2014-12-30', u'value': 10.01}, {u'date': u'2015-01-07', u'value': 10}, {u'date': u'2015-01-09', u'value': 9.97}, {u'date': u'2015-01-12', u'value': 10}, {u'date': u'2015-01-13', u'value': 10.02}, {u'date': u'2015-01-14', u'value': 10.04}, {u'date': u'2015-01-16', u'value': 10.1}, {u'date': u'2015-01-20', u'value': 10.39}, {u'date': u'2015-01-21', u'value': 9.98}, {u'date': u'2015-01-22', u'value': 9.98}, {u'date': u'2015-01-23', u'value': 9.98}, {u'date': u'2015-01-26', u'value': 10.08}, {u'date': u'2015-01-27', u'value': 10.05}, {u'date': u'2015-01-29', u'value': 10.02}, {u'date': u'2015-02-02', u'value': 10.08}, {u'date': u'2015-02-06', u'value': 9.98}, {u'date': u'2015-02-09', u'value': 9.97}, {u'date': u'2015-03-06', u'value': 10.07}, {u'date': u'2015-03-10', u'value': 10.1}, {u'date': u'2015-03-25', u'value': 10.3}, {u'date': u'2015-03-26', u'value': 10.01}, {u'date': u'2015-03-27', u'value': 10.02}, {u'date': u'2015-03-30', u'value': 10.02}, {u'date': u'2015-03-31', u'value': 10.02}, {u'date': u'2015-04-01', u'value': 10.02}, {u'date': u'2015-04-02', u'value': 10.19}, {u'date': u'2015-04-10', u'value': 10.3}, {u'date': u'2015-04-17', u'value': 10.4}, {u'date': u'2015-05-01', u'value': 10.39}, {u'date': u'2015-05-07', u'value': 10.35}, {u'date': u'2015-05-08', u'value': 10.35}, {u'date': u'2015-05-14', u'value': 10.28}, {u'date': u'2015-06-10', u'value': 10.15}, {u'date': u'2015-06-16', u'value': 10.34}, {u'date': u'2015-06-25', u'value': 10.3}, {u'date': u'2015-06-29', u'value': 10.27}, {u'date': u'2015-07-07', u'value': 10.3599}, {u'date': u'2015-07-08', u'value': 10.36}, {u'date': u'2015-07-15', u'value': 10.37}, {u'date': u'2015-07-30', u'value': 10.27}, {u'date': u'2015-08-13', u'value': 10.1}, {u'date': u'2015-08-14', u'value': 10.1}, {u'date': u'2015-08-24', u'value': 10.14}, {u'date': u'2015-08-25', u'value': 10.14}, {u'date': u'2015-08-27', u'value': 10.17}, {u'date': u'2015-08-31', u'value': 10.2}, {u'date': u'2015-09-01', u'value': 10.25}, {u'date': u'2015-09-03', u'value': 10.24}, {u'date': u'2015-09-04', u'value': 10.35}, {u'date': u'2015-09-15', u'value': 10.2401}, {u'date': u'2015-09-21', u'value': 10.22}, {u'date': u'2015-09-22', u'value': 10.22}, {u'date': u'2015-09-28', u'value': 10.2933}, {u'date': u'2015-10-02', u'value': 10.4}, {u'date': u'2015-10-06', u'value': 10.44}, {u'date': u'2015-10-12', u'value': 10.22}, {u'date': u'2015-10-13', u'value': 10.21}, {u'date': u'2015-10-22', u'value': 10.25}, {u'date': u'2015-10-28', u'value': 10.15}, {u'date': u'2015-11-04', u'value': 10.15}, {u'date': u'2015-12-09', u'value': 10}, {u'date': u'2015-12-16', u'value': 10.18}], u'priceMinDecimals': 2, u'nyTradeEndTime': u'16:30:00.000', u'lastUpdateDate': u'2015-12-16', u'nyTradeStartTime': u'09:30:00.000', u'id': u'TFSCU:US', u'timeZoneOffset': -5, u'dateTimeRanges': {}}]
The problem I found is that there exists no data for some symbols, that means the data shown on the URL is: {u'securityType': u'UNKNOWN'}. This occures for example for following symbols: AGNCB, AGNCP, MTGEP, ASRVP, AMSGP, AGIIL, ARWAU, BANFP, BKEPP, BPFHP, BBEPP.
The Problem still is how I can exclude missing data, without interrupting the loop? And why do I get an ValueError after the loop correctly has created 440 files?
The current Error:
Traceback (most recent call last):
File "<pyshell#27>", line 1, in <module>
execfile("C:\Users\Asus\Desktop\Python\web scraper\scraping\stock_to_file.py")
File "C:\Users\Asus\Desktop\Python\web scraper\scraping\stock_to_file.py", line 11, in <module>
data = json.load(htmltext)
File "C:\Python27\lib\json\__init__.py", line 291, in load
**kw)
File "C:\Python27\lib\json\__init__.py", line 339, in loads
return _default_decoder.decode(s)
File "C:\Python27\lib\json\decoder.py", line 364, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "C:\Python27\lib\json\decoder.py", line 382, in raw_decode
raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
According to your comment, the response you're getting from Bloomberg is:
{u'securityType': u'UNKNOWN'}
Bloomberg will return this if it doesn't recognize the symbol you're requesting. If you want to identify the bad symbols and continue with the script for the good ones, you can do one of:
1. Detect the error response, take any desired action, and move to the next (LBYL)
data = json.load(htmltext)
if u'securityType' in data and data[u'securityType'] == u'UNKNOWN':
print 'Unknown symbol: {0}'.format(symbol)
continue
2. Go ahead with the processing and capture the exception (EAFP)
try:
datapoints = data[0]["price"]
except KeyError:
print 'Unknown symbol: {0}'.format(symbol)
continue
It's a stylistic choice. In either case, the important thing is to continue to go to the next symbol once you detect the bad response.