We have a Django app that needs to post messages and upload files from the web server to another server via an XML API. We need to do X asynchronous file uploads and then make another XML API request when they have finished uploading. I'd also like the files to stream from disk without having to load them completely into memory first. Finally, I need to send the files as application/octet-stream in a POST body (rather than a more typical form data MIME type) and I wasn't able to find a way to do this with urllib2 or httplib.
I ended up integrating Twisted into the app. It seemed perfect for this task, and sure enough I was able to write a beautifully clean implementation with deferreds for each upload. I use my own IBaseProducer to read the data from the file in chunks and send it to the server in a POST request body. Unfortunately then I found out that the Twister reactor cannot be restarted, so I can't just run it and then stop it whenever I want to upload files. Since Twisted is apparently used more for full-blown servers, I'm now wondering whether this was the right choice.
I'm not sure if I should: a) Configure the WSGI container (currently I'm testing with manage.py) to start a Twisted thread on startup and use blockingCallFromThread to trigger my file uploads. b) Use Twisted as the WSGI container for the Django app. I'm assuming we'll want to deploy later on Apache and I'm not sure what the implications are if we take this route. c) Simply can Twisted and use some other approach for the file uploads. Kind of a shame since the Twisted approach with deferreds is elegant and works.
Which of these should we choose, or is there some other alternative?
Why would you want to deploy later on Apache? Twisted is rad. I would do (b) until someone presented specific, compelling reasons not to. Then I would do (a). Fortunately, your application code looks the same either way. blockingCallFromThread works fine whether Twisted is your WSGI container or not - either way, you're just dealing with running code in a separate thread than the reactor is running in.