I have a pool running several functions, which calls a decorated function which should append to a global list. This works properly until the pool closes, at which point the list that has been growing is empty again, for some reason.
Here is some code to reproduce my issue:
import multiprocessing as mp
import functools
foo = []
def decorator(f):
@functools.wraps(f)
def wrapper(*args, **kwargs):
foo.append(1)
print(id(foo), foo)
f(*args, **kwargs)
return wrapper
@decorator
def square(n):
return n * n
pool = mp.Pool()
pool.map(square, [i for i in range(1, 5)])
print(id(foo), foo)
This is a simplified version of what I'm actually doing. What I don't understand is why the list foo is empty after the pool.map. It's globally defined, and so each time square gets called it should have a 1 appended to it. In this code, the first 1 does get appended, but no others. That's not an issue I have in my actual codebase.
Indeed, checking the list inside the wrapping function correctly shows that the 1 has been appended, but outside it's empty again. In my actual code I get the same behavior. There, the list grows properly, however, once I access it again once the pool closes it's empty.
why the list foo is empty after the pool.map
Mutliprocessing creates processes. Two processes do not share memory, each process has his (copy) of the memory.
foo is private in each process. Because it was not modified in the starting process, it is empty.
Consider researching multiprocessing module documentation, inter-process communication methods, difference between threads and proceses, thread communication methods.