I need a task queue so that some of the heavy operations can be moved out of the uwsgi context without affecting the users. Since we're already using uwsgi app server, it'd be desirable if uwsgi spooler could be used as a task queue. I wanted to know how exactly it works. Are the spooled tasks still executed in some other uwsgi worker? If yes, then the server would still be overloaded since the other workers would be busy serving the spooled task. Are there better alternatives? I'm using python.
Reference: http://uwsgi-docs.readthedocs.org/en/latest/Spooler.html
each spooler is a different process aimed at running tasks enqueued in the form of files in a directory (the spool directory). Multiple spooler processes (per uWSGI-instance) can sit on the same spool dir to parallelize task-groups and multiple spooldirs can be configured (to have different task groups)
The spooler approach is very low-level, but requires zero-maintainance (and eventually removing tasks is a matter of rm'ing a file) and it is really solid.
The only alternative (and very probably the most used one) in the python world i am aware of is celery
otherwise you can rely on the venerable redis + daemon thread approach, where a python thread consumes tasks enqueued in redis. Eventually you can use a uWSGI mule (it is like a worker but without external access) instead of a thread to consume tasks.