I am currently creating a class which is supposed to execute some methods in a multi-threaded way, using the multiprocessing
module. I execute the real computation using a Pool
of n
workers. Now I wanted to assign each of the currently n
active workers an index between 0
and n
for some other calculation. To do this, I wanted to use a shared Queue
to assign an index in a way, that at every time no two workers have the same id. To share the same Queue
inside the class between the different threads, I wanted to store it inside a Manager.Namespace()
. But doing this, I got some problems with the Queue
. Therefore, I created a minimal version of my problem and ended up with something like this:
from multiprocess import Process, Queue, Manager, Pool, cpu_count
class A(object):
def __init__(self):
manager = Manager()
self.ns = manager.Namespace()
self.ns.q = manager.Queue()
def foo(self):
for i in range(10):
print(i)
self.ns.q.put(i)
print(self.ns.q.get())
print(self.ns.q.qsize())
a = A()
a.foo()
In this code, the execution stops before the second print statement - therefore, I think, that no data is actually written in the Queue
. When I remove the namespace
related stuff the code works flawlessly. Is this the intended behaviour of the multiprocessing
s objects and am I doing something wrong? Or is this some kind of bug?
yes, you should not use Namespace
here. when you put a Queue
object into manager.Namespace()
, each process will get a new Queue
instance, all the writer/reader of those newly created queue objects have no connection with parent process, therefore no message will be received by worker processes. share a Queue
solely instead.
by the way, you mentioned "thread" many times, but in the context of multiprocess
module, a worker is a process, not a thread.