Search code examples
pythonhttppostgetrecord

Recording HTTP in Python with Scotch

I am trying to record HTTP GET/POST requests sent by my browser using the library scotch.

I am using their sample code: http://darcs.idyll.org/~t/projects/scotch/doc/recipes.html#id2

 import scotch.proxy
 app = scotch.proxy.ProxyApp()

 import scotch.recorder
 recorder = scotch.recorder.Recorder(app, verbosity=1)

 try:

     from wsgiref.simple_server import WSGIServer, WSGIRequestHandler

     server_address = ('', 8000)
     httpd = WSGIServer(server_address, WSGIRequestHandler)

     httpd.set_app(app)

     while 1:
        httpd.handle_request()

 finally:

    from cPickle import dump
    outfp = open('recording.pickle', 'w')
    dump(recorder.record_holder, outfp)
    outfp.close()

    print 'saved %d records' % (len(recorder.record_holder))

So I ran above code, went over to google chrome, and visited a few sites to see if that would get recorded.

However, I do not see how the code should terminate. It seems that there has to be an error in httpd.handle_request() for the code to terminate.

I tried a variation of the code where I removed the try and finally syntax, and changed the while condition so that the loop ran for 30 seconds. However, that seems to be running forever as well.

Any ideas on how to get this working? I am also open to using other python libraries available for what I am trying to do: record my browser's GET/POST requests, including logons, and replay this within python.

Thanks.

Solution

  • Correct me if I'm wrong, but you're trying to log the activity of your local browser by setting a local proxy. If this is the case your browser needs to go through your proxy in order for your proxy server to log the activity.

    The code that you've provided sets a proxy server at localhost:8000, so you need to tell your browser about this. The actual setting depends on the browser, I'm sure you'd be able to google it easily.

    When I've asked to check if the code is running I actually mean whether your local proxy accepts some kind of request from the browser. Do you see the 'saved records' print out of your code at some point?