Search code examples
pythonpython-2.7python-multiprocessing

Why can't I see the queue written to in a subprocess in my main thread in Python?


I have a simple program that uses multiproessing. It basically performs a mathematical operation and writes the result to a queue. I can see the queue size grow inside the function that is multiprocessed, but once I am outside in the main thread/process, the queue is empty. It seems like some sort of scoping problem that I am not understanding. Can somebody explain why the queue is empty once outside the function? I've tried this with passing in the queue to the function as a parameter, as well as other approaches and it always seems to be empty.

from multiprocessing import Pool, Process
import math
import Queue

q = Queue.Queue(maxsize=0)


def compute_and_write(val):
    sq = val * val
    sq = math.sqrt(sq)
    sq = sq + sq + sq
    q.put("Q" + str(sq))
    print "Queue size (inside) = " + str(q.qsize())
    return sq


p = Pool(8)
y = []

for x in range(1, 100):
    y.append(x)

res = p.map(compute_and_write, y)
print "Queue size (outside) = " + str(q.qsize())

Solution

  • A Queue.Queue is no different than, e.g., a list or dict with respect to cross-process behavior: each process has its own distinct object, and changes in one process's copy have no effect at all on any other process's copy.

    You want a multiprocessing.Queue instead. That's designed to have state that's visible across processes. And that's all there should be to it.