In Python, multiple processes in a process pool encountered errors while modifying shared variables generated by the Manager module

Question

from multiprocessing import Pool, Manager

def task(args):
    k, v, sharedDict, lock = args
    with lock:
        if k not in sharedDict:
            sharedDict[k] = {}
            sharedDict[k]['current'] = v
            print(f"sharedDict[k]['current'] = {sharedDict[k]['current']}")

def main():
    manager = Manager()
    lock = manager.Lock()
    dic = manager.dict()
    pool = Pool(processes=2)
    tasks = [('a', {'A': 1}, dic, lock), ('b', {'B': 2}, dic, lock), ('c', {'C': 3}, dic, lock), ('d', {'D': 4}, dic, lock)]
    pool.map(task, tasks)
    pool.close()
    pool.join()

if __name__ == '__main__':
    main()

When I run the code above, this line throws an error:print(f"sharedDict[k]['current'] ={sharedDict[k]['current']}"),KeyError: 'current', even though I've clearly added the value to the dictionary. I hope someone can help me.

Answer 1

From Python docs1,

If standard (non-proxy) list or dict objects are contained in a referent, modifications to those mutable values will not be propagated through the manager because the proxy has no way of knowing when the values contained within are modified. However, storing a value in a container proxy (which triggers a __setitem__ on the proxy object) does propagate through the manager and so to effectively modify such an item, one could re-assign the modified value to the container proxy

sharedDict[k] = {}
sharedDict[k]['current'] = v
print(f"sharedDict[k]['current'] = {sharedDict[k]['current']}")

sharedDict[k] is non proxy nested object in in sharedDict. Its changes won't be propagated because the proxy has no way of knowing when the values contained within are modified.

The work around is to create a temporary local dict.

def task(args):
    k, v, shared_dict, lock = args
    with lock:
        if k not in shared_dict:
            shared_dict[k] = {}
            inner_dict = shared_dict[k]
            inner_dict['current'] = v
            shared_dict[k] = inner_dict
            pprint(f"{shared_dict[k]['current'] = }")

The line shared_dict[k] = inner_dict will call the __setitem__ on the sharedDict forcing it to update.

Credit to @ken