Python script is scraping the wrong page source. I think it's failing to login properly?

Question

This script succeeds at getting a 200 response object, getting a cookie, and returning reddit's stock homepage source. However, it is supposed to get the source of the "recent activity" subpage which can only be accessed after logging in. This makes me think it's failing to log in appropriately but the username and password are accurate, I've double checked that.

#!/usr/bin/python
import requests
import urllib2

auth = ('username', 'password')
with requests.session(auth=auth) as s:
    c = s.get('http://www.reddit.com')
    cookies = c.cookies
    for k, v in cookies.items():
            opener = urllib2.build_opener()
            opener.addheaders.append(('cookie', '{}={}'.format(k, v)))
            f = opener.open('http://www.reddit.com/account-activity')
            print f.read()

Answer 1

It looks like you're using the standard "HTTP Basic" authentication, which is not what Reddit uses to log in to its web site. (Almost no web sites use HTTP Basic (which pops up a modal dialog box requesting authentication), but implement their own username/password form).

What you'll need to do is get the home page, read the login form fields, fill in the user name and password, POST the response back to the web site, get the resulting cookie, then use the cookie in future requests. There may be quite a number of other details for you to work out too, but you'll have to experiment.