Search code examples
pythonvariablesmultiprocessingglobal

Global variables in multiprocessing (Python)


I have made simple (Python) code example of my common question. For multiprocessing code I need execute def inside of def for each processors. If use only one def (def f) - the result is ok (I can counting variables globally because use manager for it). But if use two level of def (def ff) - result fail. Any change in def ff not apply in def f later.

from multiprocessing import Process, Manager
import os

def ff(b):
    b = b +1
    print('def ff b = ', b)
    print('parent process:', os.getppid())
    print('process id:', os.getpid())
    print()

def f(a):
    b = 0
    ff(b)
    a.value = a.value + b
    print('def f a = ', a.value, ' b = ', b)
    print('parent process:', os.getppid())
    print('process id:', os.getpid())
    print()

if __name__ == '__main__':
    #a = ()
    manager = Manager()
    a = manager.Value('i', 0)
    p = Process(target=f, args=(a,))
    p.start()
    p.join()
    print('Main, a = ', a.value)
    print('parent process:', os.getppid())
    print('process id:', os.getpid())

this is result

def ff b =  1
parent process: 12312
process id: 2320

def f a =  0  b =  0
parent process: 12312
process id: 2320

Main, a =  0
parent process: 21296
process id: 12312

My expectation:

def f return b = 1 and a = 1
Main return a = 1

What I made wrong? How to make variables inside of processing Global?


Solution

  • You expected b to become 1 in f, and therefore a to become 1 in the parent. Your problem has nothing to do with multiprocessing or globals, you've just misunderstood the argument passing conventions of Python. The issue you're having is that you can't mutate b in f through changes in a function, ff, it's passed to; ints are immutable, and you can't pass a reference to a name to a function such that the caller's name can be rebound.

    Fixing your code is trivial; instead of trying to do C++-style pass-by-reference to achieve the change to b (which Python can't do), you need to return the new value (comments on changed/added lines):

    def ff(b):
        b = b +1
        print('def ff b = ', b)
        print('parent process:', os.getppid())
        print('process id:', os.getpid())
        print()
        return b   # Return new b
    
    def f(a):
        b = 0
        b = ff(b)  # Assign returned value back to b
        a.value = a.value + b
        print('def f a = ', a.value, ' b = ', b)
        print('parent process:', os.getppid())
        print('process id:', os.getpid())
        print()
    

    That's it. All the rest of what you did works.