Search code examples
pythonmultithreadingconcurrencyqueuegevent

Why is there `gevent.queue.Queue` if `queue.Queue` already exists and is thread-safe?


There is this question Why do we need gevent.queue? which asks about the necessity of gevent.queue.Queue regarding the fact that there is no parallelism, but only concurrency with gevent. Rightfully, the answer explains:

[...] you might want to ensure that a series of statements execute atomically [...]

So each method should be atomic, which explains why we need a queue object instead of a "regular" list, for example. However, according to Python docs https://docs.python.org/3/library/queue.html a queue.Queue object already has those safety features:

It is especially useful in threaded programming when information must be exchanged safely between multiple threads. The Queue class in this module implements all the required locking semantics.

So why is there gevent.queue.Queue when queue.Queue already exists? Are more safety features required or is it a question of performance?


Solution

  • There is an answer to this question in this comment in one of the GitHub issues:

    gevent's Queue is directly implemented in terms of gevent primitives, rather than going through the abstraction of the monkey-patched threading libraries like the standard library Queue must do, and gevent's Queue can take advantage of the fact that certain operations cannot be interrupted by other greenlets. This lets it have reduced overhead and better performance.

    gevent's Queue is compatible with gevent and greenlets even if monkey-patching isn't used, whereas the standard library Queue isn't and requires monkey-patching. Not everyone like to monkey-patch.

    gevent's Queue also has a slightly richer interface, for example it can be iterated.