How do I unit test a module that relies on urllib2?

Question

I've got a piece of code that I can't figure out how to unit test! The module pulls content from external XML feeds (twitter, flickr, youtube, etc.) with urllib2. Here's some pseudo-code for it:

params = (url, urlencode(data),) if data else (url,)
req = Request(*params)
response = urlopen(req)
#check headers, content-length, etc...
#parse the response XML with lxml...

My first thought was to pickle the response and load it for testing, but apparently urllib's response object is unserializable (it raises an exception).

Just saving the XML from the response body isn't ideal, because my code uses the header information too. It's designed to act on a response object.

And of course, relying on an external source for data in a unit test is a horrible idea.

So how do I write a unit test for this?

Answer 1

urllib2 has a functions called build_opener() and install_opener() which you should use to mock the behaviour of urlopen()

import urllib2
from StringIO import StringIO

def mock_response(req):
    if req.get_full_url() == "http://example.com":
        resp = urllib2.addinfourl(StringIO("mock file"), "mock message", req.get_full_url())
        resp.code = 200
        resp.msg = "OK"
        return resp

class MyHTTPHandler(urllib2.HTTPHandler):
    def http_open(self, req):
        print "mock opener"
        return mock_response(req)

my_opener = urllib2.build_opener(MyHTTPHandler)
urllib2.install_opener(my_opener)

response=urllib2.urlopen("http://example.com")
print response.read()
print response.code
print response.msg