passing dict of dict between process

Question

I can't pass a dict of dict between processes with multiprocessing in Python2.7

from multiprocessing import Process, Manager

class TestProcess(Process):
    def __init__(self, d, w):
        super(TestProcess, self).__init__()
        self.daemon = True
        self.d = d
        self.w = w

    def run(self):
        self.d['a'][self.w] = self.w


class Test(object):
    def __init__(self):
        manager = Manager()
        self.d = manager.dict()
        self.d['a'] = {}

        p = TestProcess(d=self.d, w=1)
        p.start()
        p.join()

        p = TestProcess(d=self.d, w=2)
        p.start()
        p.join()

    def show(self):
        print(self.d)

if __name__ == '__main__':

    t=Test()
    t.show()

Expecting result: {'a': {1:1, 2:2}}, but got {'a': {}} instead

Answer 1

Make the nested dictionary a proxy object too:

manager = Manager()
self.d = manager.dict()
self.d['a'] = manager.dict() 

or retrieve self.d['a'], mutate that object, and set it back:

a = self.d['a']
a[self.w] = self.w
self.d['a'] = a

self.d is not a regular dictionary; it wraps a dict to track mutations to the object itself, but your value, associated with the key 'a' is not such an object, so mutations to that object are not detected, and the changes are not communicated with the parent process.

If you were to use self.d['a'][self.w] = self.w, then self.d['a'] = self.d['a'] the latter would retrieve the still empty dictionary from the parent process and set self.d['a'] back to that empty object; you have to create a local reference to the remote mutable nested dictionary first to ensure you correctly tell the parent process to correctly set it.

From the Proxy Objects section of the documentation:

If standard (non-proxy) list or dict objects are contained in a referent, modifications to those mutable values will not be propagated through the manager because the proxy has no way of knowing when the values contained within are modified. However, storing a value in a container proxy (which triggers a __setitem__ on the proxy object) does propagate through the manager and so to effectively modify such an item, one could re-assign the modified value to the container proxy:

# create a list proxy and append a mutable object (a dictionary)
lproxy = manager.list()
lproxy.append({})
# now mutate the dictionary
d = lproxy[0]
d['a'] = 1
d['b'] = 2
# at this point, the changes to d are not yet synced, but by
# updating the dictionary, the proxy is notified of the change
lproxy[0] = d