Search code examples
pythonlinuxpython-2.7ubuntupycurl

Retrieve the source code of a url using pycurl and a port number?


Is there any way to retrieve the url source code and store it in a string ,provided that a particular port is defined in pycurl module so that it works for a proxy network. platform - ubuntu or any other linux distro


Solution

  • Use this code to get source of url

    from StringIO import StringIO    
    import pycurl
    
    url = 'http://www.google.com/'
    
    storage = StringIO()
    c = pycurl.Curl()
    c.setopt(c.URL, url)
    c.setopt(c.WRITEFUNCTION, storage.write)
    c.perform()
    c.close()
    content = storage.getvalue()
    print content