Search code examples
python-3.xweb-scrapingurlliburllib3

urllib3 return 404 not found for existing website


Different result by urllib and urllib3

I can open a web page by copying the address into my chrome and urllib also returns the page source code. I just do not understand why urllib3 is returning 404 not found on this webpage when everything else works.

Below is the original code:

url = 'http://www.webmd.com/drugs/2/condition-12862/depression%20associated%20with%20bipolar%20disorder'
import urllib3
http = urllib3.PoolManager()
r = http.request('GET',url)
r.data

import urllib.request
req = urllib.request.Request(url=url)
with urllib.request.urlopen(req) as f:
    print(f.read().decode('utf-8'))

Solution

  • My guess, you are calling behind proxy, urllib uses system proxy (if you are on linux - http_proxy enviroment variable), for urllib3 you need to specify it using urllib3 library