I am trying to get the final url of a page that seems to redirect more than once. Try this sample URL in your browser and compare it to the final URL at the bottom of my code snippet:
Link that redirects more than once
And here is the test code I was running, notice the final URL that gets a code of 200 isn't the same as the one in your browser. What are my options?
Python 2.7.1+ (r271:86832, Apr 11 2011, 18:13:53)
[GCC 4.5.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import httplib
>>> from urlparse import urlparse
>>> url = 'http://www.usmc.mil/units/hqmc/'
>>> host = urlparse(url)[1]
>>> req = ''.join(urlparse(url)[2:5])
>>> conn = httplib.HTTPConnection(host)
>>> conn.request('HEAD', req)
>>> resp = conn.getresponse()
>>> print resp.status
301
>>> print resp.msg.dict['location']
http://www.marines.mil/units/hqmc/
>>> url = 'http://www.marines.mil/units/hqmc/'
>>> host = urlparse(url)[1]
>>> req = ''.join(urlparse(url)[2:5])
>>> conn = httplib.HTTPConnection(host)
>>> conn.request('HEAD', req)
>>> resp = conn.getresponse()
>>> print resp.status
302
>>> print resp.msg.dict['location']
http://www.marines.mil/units/hqmc/default.aspx
>>> url = 'http://www.marines.mil/units/hqmc/default.aspx'
>>> host = urlparse(url)[1]
>>> req = ''.join(urlparse(url)[2:5])
>>> conn = httplib.HTTPConnection(host)
>>> conn.request('HEAD', req)
>>> resp = conn.getresponse()
>>> print resp.status
200
>>> print resp.msg.dict['location']
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
KeyError: 'location'
>>> print url
http://www.marines.mil/units/hqmc/default.aspx //THIS URL DOES NOT RETURN A 200 IN ANY BROWSER I HAVE TRIED
you can try to set your User-Agent header to the User-Agent of your browser.
ps: urllib2 automatically redirects
EDIT:
In [2]: import urllib2
In [3]: resp = urllib2.urlopen('http://www.usmc.mil/units/hqmc/')
In [4]: resp.geturl()
Out[4]: 'http://www.marines.mil/units/hqmc/default.aspx