Search code examples
pythonconcurrencygevent

gevent queue does not raise Empty when queue.get invoked inside greenlets (other than the main greenlet)?


I am very new to gevent and my question might even be very naive, but the when I run following code I get queue empty exception.

case 1 makes sense

import gevent
from gevent import monkey
monkey.patch_all()

q = gevent.queue.Queue(maxsize=10)

try:
    q.get(timeout=1)
except gevent.queue.Empty:
    print("main greenlet queue empty")

but if I run following code I do not get any timeout and code just never invokes any exceptions and I never enter the exception handeling block, in fact code doesn't even wait for 1 sec to fetch data from queue. I just executes immediately with the shown output.

case 2

import gevent
from gevent import monkey
monkey.patch_all()

q = gevent.queue.Queue(maxsize=10)

def worker(idx):
    print("worker", idx)
    try:
        q.get(timeout=1)
    except gevent.queue.Empty:
        print("worker", idx, "queue empty")

workers = [gevent.spawn(worker, i) for i in range(1)]

this outputs:

worker 0

However, if I call get in the main greenlet with a call to queue.get in both child and main greenlet like following, the behaviour now changes back to normal.

case 3

import gevent
from gevent import monkey
monkey.patch_all()

q = gevent.queue.Queue(maxsize=10)

def worker(idx):
    print("worker", idx)
    try:
        q.get(timeout=1)
    except gevent.queue.Empty:
        print("worker", idx, "queue empty")

workers = [gevent.spawn(worker, i) for i in range(1)]

try:
    q.get(timeout=1)
except gevent.queue.Empty:
    print("main greenlet queue empty")

I get following output.

worker 0
main greenlet queue empty
worker 0 queue empty

I fail to understand that why does case 2 not raise an exception, can some one please explain to me what am I doing wrong?

I am using python 3.7 and gevent '21.1.2'


Solution

  • In your second case, you are starting greenlets (the interpreter exits after). Adding joinall() makes your code wait for the spawned greenlets to return:

    q = gevent.queue.Queue(maxsize=10)
    
    def worker(idx):
        print("worker", idx)
        try:
            q.get(timeout=1)
        except gevent.queue.Empty:
            print("worker", idx, "queue empty")
    
    
    print(q.empty())
    workers = [gevent.spawn(worker, i) for i in range(1)]
    # waiting for them to finish, works as expected!
    gevent.joinall(workers)
    

    Out:

    True
    worker 0
    worker 0 queue empty