Search code examples
pythonpython-3.xweb-scrapingurllib

python error: urlopen error [Errno 11001] getaddrinfo failed


We are using proxy.

I am able to open http://www.google.com and www.bing.com with the codes below.

But, I am getting error "urlopen error [Errno 11001] getaddrinfo failed" when trying to open other websites, for example: https://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/2.5_day.geojson, https://www.pythonprogramming.net, www.aa.com, etc.

The rules for http and https are the same, ie http://MyCompanyProxy:Port/.

How can I set both http and https rules to be http://MyCompanyProxy:Port/ ?

Thank you.

import urllib.request 

def main():
 urlData = "http://google.com/"
 ***edited****
 proxy = urllib.request.ProxyHandler({'http': 'http://MyCompanyProxy:Port/','https': 'http://MyCompanyProxy:Port/'})
 ****end of edit*******
 auth = urllib.request.HTTPBasicAuthHandler()
 opener = urllib.request.build_opener(proxy, auth, urllib.request.HTTPHandler)
 urllib.request.install_opener(opener)

 webUrl = urllib.request.urlopen(urlData)              #--->> error here
 print ("result code: " + str(webUrl.getcode()))

if __name__ == "__main__":
  main()

Solution

  • You should call the ProxyHandler with an 'https' rule, too. Not only 'http' if you want to access