I want to get elements from a multiprocessing.Queue only if it has at least N items:
def popn(n, q, lock):
lock.acquire()
ret = []
if q.qsize() >= n:
for i in range(n):
ret.append(q.get())
lock.release()
return ret
def push(element, q, lock):
lock.acquire()
q.put(element)
lock.release()
However, according to docs, multiprocessing.Queue.qsize() is not reliable, I want to know whether it will be accurate size of the queue if I protect the queue with multiprocessing.Lock(as is shown in the code above)?
Kinda... but your code won't work on Mac, and on other platforms, you might deadlock.
First, Mac doesn't implement the sem_getvalue function needed to support multiprocessing.Queue.qsize(). On Mac, your code will fail with a NotImplementedError.
Second, if a queue is too full, q.put can block until something calls get. If you try to protect the queue with a lock, then it's impossible for that get call to happen, because the lock is being held by the process stuck in the middle of put. (This can happen even if you don't pass maxsize, although you're unlikely to run into this behavior without passing maxsize - the queue size would have to hit the semaphore implementation's SEM_VALUE_MAX, which I believe is very high on most systems.)