Downloading a webpage using urllib3

Question

I'm trying to write a program for an assignment that uses urllib3 to download a webpage and store it in a dictionary. (I'm using spyder 3.6) The program is giving me an 'AttributeError' and I have no idea what I'm doing wrong. here is my code with step by step notes I wrote for the assignment.

#Downloading a webpage
import urllib3
import sys
#these import statements allow us to use 'modules' aka 'libraries' ....
#code written by others that we can use

urlToRead = 'http://www.google.com'
#This value won't actually get used, because of the way the while loop
#below is set up. But while loops often need a dummy value like this to
#work right the first time

crawledWebLinks = {}
#Initialize an empty dictionary, in which (key, value) pairs will correspond to (short, url) eg
#("Goolge" , "http://www.google.com")

#Ok, there is a while loop coming up

#Here ends the set up

while urlToRead != ' ':
#This is a condition that dictates that the while loop will keep checking
#as long as this condition is true the loop will continue, if false it will stop
    try:
        urlToRead = input("Please enter the next URL to crawl")
    #the "try" prevents the program from crashing if there is an error
    #if there is an error the program will be sent to the except block
        if urlToRead == '':
            print ("OK, exiting loop")
            break
        #if the user leaves the input blank it will break out of the loop
        shortName = input("Please enter a short name for the URL " + urlToRead)
        webFile = urllib3.urlopen(urlToRead).read()
        #This line above uses a ready a readymade function in the urllib3 module to
        #do something super - cool:
        #IT takes a url, goes to the website for the url, downloads the
        #contents (which are in the form of HTML) and returns them to be
        #stored in a string variable (here called webFile)
        crawledWebLinks[shortName] = webFile
        #this line above place a key value pair (shortname, HTML for that url)
        #in the dictionary
    except:
        #this bit of code - the indented lines following 'except:' will be
        #excecuted if the code in the try block (the indented following lines
        #the 'try:' above) throw and error
        #this is an example of something known as exeption-handling
        print ("*************\nUnexpected Error*****", sys.exc_info()[0])
        #The snip 'sys.exc_info()[0]' return information about the last
        #error that occurred - 
        #this code is made available through the sys library that we imported above
        #Quite Magical :)
        stopOrProceed = input("Hmm..stop or proceed? Enter 1 to stop, enter anything else to continue")
        if stopOrProceed ==1 :
            print ('OK...Stopping\n')
            break
        #this break will break out of the nearest loop - in this case,
        #the while loop
    else:
        print ("Cool! Let's continue\n")
        continue
        # this continue will skip out of the current iteration of this 
        #loop and move to the next i.e. the loop will reset to the start
print (crawledWebLinks.keys())

Answer 1

Your issue is that you are trying to call urllib3.urlopen(), and urllib3 does not have a member urlopen Here is a working snippet. All that I did was replace urllib3 with urllib.request:

import urllib.request
import sys

urlToRead = 'http://www.google.com'

crawledWebLinks = {}

while urlToRead != ' ':
    try:
        urlToRead = input("Please enter the next URL to crawl: ")
        if urlToRead == '':
            print ("OK, exiting loop")
            break
        #if the user leaves the input blank it will break out of the loop
        shortName = input("Please enter a short name for the URL " + urlToRead + ": ")
        webFile = urllib.request.urlopen(urlToRead).read()
        crawledWebLinks[shortName] = webFile
    except:
        print ("*************\nUnexpected Error*****", sys.exc_info()[0])
        stopOrProceed = input("Hmm..stop or proceed? Enter 1 to stop, enter anything else to continue")
        if stopOrProceed ==1 :
            print ('OK...Stopping\n')
            break
    else:
        print ("Cool! Let's continue\n")
        continue
print (crawledWebLinks)

Another note, simply printing out the type of error in your except block is not very useful. I was able to debug your code in 30 seconds once I removed that and viewed the actual traceback.