Search code examples
pythonpython-2.7beautifulsoupurllib2

opening link from variable Python (urllib2.urlopen) + Beautifulsoup4

I am working with Python 2.7 + urllib2 + Beautifulsoup4

When I have string:

soup = BeautifulSoup(urllib2.urlopen('http://www.some-website.com', 'html'))

It works perfectly, but when I move URl to variable, it is not working.

variable = 'http://www.some-website.com'
soup = BeautifulSoup(urllib2.urlopen(variable, 'html'))

error:

edit: errcode is: File "C:\Python27\lib\urllib2.py", line 285, in get_type
  raise ValueError, "unknown url type: %s" % self.__original
    ValueError: unknown url type: api/Abc-Abc/def/7/179 –

Solved

problem was that one of the links was only a reference to the server's database.

Solution 
  • # Note: Make sure you add live website like http://vaibhavmule.com not http://some-website.com
variable = 'http://www.some-website.com' # Do not forget 'http' prefix here

# As you used 'html' which is not parser library.
soup = BeautifulSoup(urllib2.urlopen(variable))

    This should work.

    Reference for using parser library.