This code for downloading a json file from a url runs alright when I type commands one by one in the python shell. However, when I try to run the module containing this code I get: ValueError: No JSON object could be decoded. Any ideas why is that? I run python 2.7.
import urllib2
from urllib2 import Request
import json
import re
url1 = "http://www.skyscanner.net/flights/lond/nyca/130514/130525/airfares-from-london-to-new-york-in-may-2013.html"
req = Request(url1)
res = urllib2.urlopen(req)
the_page = res.read()
theText = str(the_page)
myre = re.compile(r'"SessionKey":"((([a-z0-9]+-)+)[a-z0-9]{12})"')
match = re.search(myre, theText)
print match.group(1)
url2 = "http://www.skyscanner.net/dataservices/routedate/v2.0/"+str(match.group(1))
htmltext = urllib2.urlopen(url2)
data = json.load(htmltext)
the whole code now:
import urllib2
from urllib2 import Request, urlopen, URLError, HTTPError
import json
import re
url1 = "http://www.skyscanner.net/flights/lond/nyca/130514/130525/airfares-from-london-to-new-york-in-may-2013.html"
req = Request(url1)
res = urllib2.urlopen(req)
the_page = res.read()
theText = str(the_page)
myre = re.compile(r'"SessionKey":"((([a-z0-9]+-)+)[a-z0-9]{12})"')
match = re.search(myre, theText)
url2 = "http://www.skyscanner.net/dataservices/routedate/v2.0/%s" % str(match.group(1))
req2 = urllib2.Request(url2)
try:
response = urlopen(req2)
except HTTPError as e:
print 'The server couldn\'t fulfill the request.'
print 'Error code: ', e.code
except URLError as e:
print 'We failed to reach a server.'
print 'Reason: ', e.reason
else:
data = json.loads(response.read())
print data["SessionKey"]
Ha! You were correct, it is a difference between shell use and scripted use. Whatever that url1 is doing, it hasn't completed (remotely) by the time you open url2. Putting a short wait time in makes it all work!
import time
url2 = "http://www.skyscanner.net/dataservices/routedate/v2.0/%s" % str(match.group(1))
time.sleep(5)
req2 = urllib2.Request(url2)
so it wasn't your code at all... ho hum!