How can I write tests for code using twisted.web.client.Agent and its subclasses?

I read the official tutorial on test-driven development, but it hasn't been very helpful in my case. I've written a small library that makes extensive use of twisted.web.client.Agent and its subclasses (BrowserLikeRedirectAgent, for instance), but I've been struggling in adapting the tutorial's code to my own test cases.

I had a look at twisted.web.test.test_web, but I don't understand how to make all the pieces fit together. For instance, I still have no idea how to get a Protocol object from an Agent, as per the official tutorial

Can anybody show me how to write a simple test for some code that relies on Agent to GET and POST data? Any additional details or advice is most welcome...

Many thanks!

Solution

How about making life simpler (i.e. code more readable) by using @inlineCallbacks.

In fact, I'd even go as far as to suggest staying away from using Deferreds directly, unless absolutely necessary for performance or in a specific use case, and instead always sticking to @inlineCallbacks—this way you'll keep your code looking like normal code, while benefitting from non-blocking behavior:

from twisted.internet import reactor
from twisted.web.client import Agent
from twisted.internet.defer import inlineCallbacks
from twisted.trial import unittest
from twisted.web.http_headers import Headers
from twisted.internet.error import DNSLookupError


class SomeTestCase(unittest.TestCase):
    @inlineCallbacks
    def test_smth(self):
        ag = Agent(reactor)
        response = yield ag.request('GET', 'http://example.com/', Headers({'User-Agent': ['Twisted Web Client Example']}), None)
        self.assertEquals(response.code, 200)

    @inlineCallbacks
    def test_exception(self):
        ag = Agent(reactor)
        try:
            yield ag.request('GET', 'http://exampleeee.com/', Headers({'User-Agent': ['Twisted Web Client Example']}), None)
        except DNSLookupError:
            pass
        else:
            self.fail()

Trial should take care of the rest (i.e. waiting on the Deferreds returned from the test functions (@inlineCallbacks-wrapped callables also "magically" return a Deferred—I strongly suggest reading more on @inlineCallbacks if you're not familiar with it yet).

P.S. there's also a Twisted "plugin" for nosetests that enables you to return Deferreds from your test functions and have nose wait until they are fired before exiting: http://nose.readthedocs.org/en/latest/api/twistedtools.html