Search code examples
pythontwisted

Compound custom service server using Twisted

I have an interesting project going on at our workplace. The task, that stands before us, is such:

  • Build a custom server using Python
  • It has a web server part, serving REST
  • It has a FTP server part, serving files
  • It has a SMTP part, which receives mail only
  • and last but not least, a it has a background worker that manages lowlevel file IO based on requests received from the above mentioned services

Obviously the go to place was Twisted library/framework, which is an excelent networking tool. However, studying the docs further, a few things came up that I'm not sure about.

Having Java background, I would solve the task (at least at the beginning) by spawning a separate thread for each service and going from there. Being in Python however, I cannot do that for any reasonable purpose as Python has GIL. I'm not sure, how Twisted handles this. I would expect, that Twisted has large (if not majority) code written in C, where GIL is not the issue, but that I couldn't find the docs explained to my satisfaction.

So the most oustanding question is: Given that Twisted uses Reactor as it's main design pattern, will it be able to:

  1. Serve all those services needed
  2. Do it in a non-blocking fashion (it should, according to docs, but if someone could elaborate, I'd be grateful)
  3. Be able to serve about few hundreds of clients at once
  4. Serve large file downloads in a reasonable way, meaning that it can serve multiple clients, using multiple services, downloading and uploading large files.

Large files being in the order of hundres of MB, or few GB. The size is not important, it's the time that the client has to stay connected to the server that matters.

Edit: I'm actually inclined to go the way of python multiprocessing, but not sure, whether that's a correct thing to do with Twisted etc.

Solution
    • Serve all those services needed

    Yes.

    • Do it in a non-blocking fashion (it should, according to docs, but if someone could elaborate, I'd be grateful)

    Twisted's uses the common reactor model. I/O goes through your choice of poll, select, whatever to determine if data is available. It handles only what is available, and passes the data along to other stages of your app. This is how it is non-blocking.

    I don't think it provides non-blocking disk I/O, but I'm not sure. That feature not what most people need when they say non-blocking.

    • Be able to serve about few hundreds of clients at once

    Yes. No. Maybe. What are those clients doing? Is each hitting refresh every second on a browser making 100 requests? Is each one doing a numerical simulation of galaxy collisions? Is each sending the string "hi!" to the server, without expecting a response?

    Twisted can easily handle 1000+ requests per second.

    • Serve large file downloads in a reasonable way, meaning that it can serve multiple clients, using multiple services, downloading and uploading large files.

    Sure. For example, the original version of BitTorrent was written in Twisted.