Search code examples
pythonmultiprocessing

Managed dictionary does not behave as expected in multiprocessing


From what I've managed to figure out so far, this may only be a problem in macOS.

Here's my MRE:

from multiprocessing import Pool, Manager
from functools import partial

def foo(d, n):
    d.setdefault("X", []).append(n)

def main():
    with Manager() as manager:
        d = manager.dict()
        with Pool() as pool:
            pool.map(partial(foo, d), range(5))
        print(d)

if __name__ == "__main__":
    main()

Output:

{'X': []}

Expected output:

{'X': [0, 1, 2, 3, 4]}

Platform:

macOS 14.3.1
Python 3.12.2

Maybe I'm doing something fundamentally wrong but I understood that the whole point of the Manager was to handle precisely this kind of scenario.

EDIT

There is another hack which IMHO should be unnecessary but even this doesn't work (produces identical output):

from multiprocessing import Pool, Manager

def ipp(d, lock):
    global gDict, gLock
    gDict = d
    gLock = lock

def foo(n):
    global gDict, gLock
    with gLock:
        gDict.setdefault("X", []).append(n)

def main():
    with Manager() as manager:
        d = manager.dict()
        lock = manager.Lock()
        with Pool(initializer=ipp, initargs=(d, lock)) as pool:
            pool.map(foo, range(5))
        print(d)

if __name__ == "__main__":
    main()

Solution

  • This is a crappy "answer" and I apologize for that and will delete it once you post a comment that you have seen it. I get the distinct impression that the answer is related to the "sub-list" not being managed per that post I mentioned.

    For example:

    This seems to work:

    def foo(d, n):
        d.setdefault("X", []).append(n) # the thing being updated is managed
    
    def main():
        with Manager() as manager:
            d = manager.dict()
            d["X"] = manager.list()
            with Pool() as pool:
                pool.map(partial(foo, d), range(5))
            print(list(d["X"]))
    
    if __name__ == "__main__":
        main()
    

    as does:

    def foo(d, n):
        target = d.setdefault("X", [])
        target.append(n)
        d["X"] = target  # the thing being updated is managed
    
    def main():
        with Manager() as manager:
            d = manager.dict()
            with Pool() as pool:
                pool.map(partial(foo, d), range(5))
            print(d)
    
    if __name__ == "__main__":
        main()
    

    So updating a "managed" object seems to work while updating an unmanaged list even inside a managed container object does not.