I have a simple program that uses multiproessing. It basically performs a mathematical operation and writes the result to a queue. I can see the queue size grow inside the function that is multiprocessed, but once I am outside in the main thread/process, the queue is empty. It seems like some sort of scoping problem that I am not understanding. Can somebody explain why the queue is empty once outside the function? I've tried this with passing in the queue to the function as a parameter, as well as other approaches and it always seems to be empty.
from multiprocessing import Pool, Process
import math
import Queue
q = Queue.Queue(maxsize=0)
def compute_and_write(val):
sq = val * val
sq = math.sqrt(sq)
sq = sq + sq + sq
q.put("Q" + str(sq))
print "Queue size (inside) = " + str(q.qsize())
return sq
p = Pool(8)
y = []
for x in range(1, 100):
y.append(x)
res = p.map(compute_and_write, y)
print "Queue size (outside) = " + str(q.qsize())
A Queue.Queue
is no different than, e.g., a list or dict with respect to cross-process behavior: each process has its own distinct object, and changes in one process's copy have no effect at all on any other process's copy.
You want a multiprocessing.Queue
instead. That's designed to have state that's visible across processes. And that's all there should be to it.