From what I've managed to figure out so far, this may only be a problem in macOS.
Here's my MRE:
from multiprocessing import Pool, Manager
from functools import partial
def foo(d, n):
d.setdefault("X", []).append(n)
def main():
with Manager() as manager:
d = manager.dict()
with Pool() as pool:
pool.map(partial(foo, d), range(5))
print(d)
if __name__ == "__main__":
main()
Output:
{'X': []}
Expected output:
{'X': [0, 1, 2, 3, 4]}
Platform:
macOS 14.3.1
Python 3.12.2
Maybe I'm doing something fundamentally wrong but I understood that the whole point of the Manager was to handle precisely this kind of scenario.
EDIT
There is another hack which IMHO should be unnecessary but even this doesn't work (produces identical output):
from multiprocessing import Pool, Manager
def ipp(d, lock):
global gDict, gLock
gDict = d
gLock = lock
def foo(n):
global gDict, gLock
with gLock:
gDict.setdefault("X", []).append(n)
def main():
with Manager() as manager:
d = manager.dict()
lock = manager.Lock()
with Pool(initializer=ipp, initargs=(d, lock)) as pool:
pool.map(foo, range(5))
print(d)
if __name__ == "__main__":
main()
This is a crappy "answer" and I apologize for that and will delete it once you post a comment that you have seen it. I get the distinct impression that the answer is related to the "sub-list" not being managed per that post I mentioned.
For example:
This seems to work:
def foo(d, n):
d.setdefault("X", []).append(n) # the thing being updated is managed
def main():
with Manager() as manager:
d = manager.dict()
d["X"] = manager.list()
with Pool() as pool:
pool.map(partial(foo, d), range(5))
print(list(d["X"]))
if __name__ == "__main__":
main()
as does:
def foo(d, n):
target = d.setdefault("X", [])
target.append(n)
d["X"] = target # the thing being updated is managed
def main():
with Manager() as manager:
d = manager.dict()
with Pool() as pool:
pool.map(partial(foo, d), range(5))
print(d)
if __name__ == "__main__":
main()
So updating a "managed" object seems to work while updating an unmanaged list even inside a managed container object does not.