I'm developing a system where a queue will be filled with millions of items
I need a process that reads items from the queue constantly and then sends those items out to registered clients.
I'm thinking about using twisted for this, having the queue reader be a twisted server listening on a tcp port then clients can connect on that port and when an item is pulled from the queue the server writes it out to all the clients.
Does that sound like something that twisted would be ideal for? Does anyone know of any sample code out there that may do something similar?
Perspective Broker might be what you're looking for. If you dont know PB, think of it as XML-RPC but much more. You can invoke remote procedures (duh!), pass most of python objects around just to name two.
For your case, have a PB for clients. Twisted Server can invoke the remote doWork(myItem, myParameters). The Twisted Server will be a persistent process running which accumulates items in a queue. If you have millions (or even server tens) items, keep in mind of gracefully recovering in case of a crash. How about smartly backing stuff onto disk every now and then and reading this up when starting up ?