from concurrent import futures
class MyClass:
def __init__(self):
self.mylist = []
def test(self, i):
self.mylist.append(i)
myclass = MyClass()
print(myclass.mylist)
ilist = [1, 2, 3, 4]
for i in ilist:
myclass.test(i)
print(myclass.mylist)
myclass.mylist = []
with futures.ProcessPoolExecutor() as pool:
for null in pool.map(myclass.test, ilist):
pass
print(myclass.mylist)
Output:
[]
[1, 2, 3, 4]
[]
Why does appending values in def test to self.mylist work in conventional loop but not when using futures? How to allow append in function while using futures?
Lets tweak the program a bit so that the function executed by the pool returns the list and also lets print the address of the MyClass object.
from concurrent import futures
class MyClass:
def __init__(self):
self.mylist = []
def test(self, i):
print(hex(id(self)), self.mylist, i)
self.mylist.append(i)
return self.mylist
if __name__ == "__main__":
myclass = MyClass()
ilist = [1, 2, 3, 4]
myclass.mylist = []
with futures.ProcessPoolExecutor() as pool:
for null in pool.map(myclass.test, ilist):
print(f'Output of process: {null}')
print(f'addr: {hex(id(myclass))} , {myclass.mylist}')
gives the output
Output of process: [1]
Output of process: [2]
Output of process: [3]
Output of process: [4]
0x1b88e358860 [] 1
0x20bffa28908 [] 3
0x259844b87f0 [] 2
0x1d7546d8898 [] 4
addr: 0x20e5ebc5400 , []
As you could see each process is handling a different copy of the MyClass object.
Lets now replace the ProcessPoolExecutor with ThreadPoolExecutor.
Now the result looks like this:
0x1a323eb5438 [] 1
0x1a323eb5438 [1] 2
0x1a323eb5438 [1, 2] 3
0x1a323eb5438 [1, 2, 3] 4
Output of process: [1, 2, 3, 4]
Output of process: [1, 2, 3, 4]
Output of process: [1, 2, 3, 4]
Output of process: [1, 2, 3, 4]
addr: 0x1a323eb5438 , [1, 2, 3, 4]
Now each thread is dealing with the same object.
In short, processes have their own memory and is not shared across the processes.