Multiprocessing: cannot share a variable between files

Question

I describe my issue with a minimal example. In my project a config file contains a variable param which is used by all other modules:

# config.py
param = -1

The config is imported in solver which performs a computation based on the passed val and the param from config:

# solver.py
import config

def solve(val):
    # some computation using val and config.param
    print('solver uses param:', config.param)
    return None

My goal is to perform the computation for some param and various vals.

# main.py
import multiprocessing as mp
import config, solver

val_list = [None for i in range(3)]
pool = mp.Pool(10)
config.param = 1
print('testing param: ', config.param)
results = pool.map(solver.solve, [val for val in val_list]) # multiprocessing
#results = [solver.solve(val) for val in val_list]          # basic

While using multiprocessing I get an output:

testing param:  1
solver uses param: -1
solver uses param: -1
solver uses param: -1

which means that the change of param in main isn't seen by solver. However, it is seen when using basic for-loop computation:

testing param:  1
solver uses param: 1
solver uses param: 1
solver uses param: 1

How can I get the desired behavior with multiprocessing? Note that the change of param shouldn't be done manually in config, as in reality I test different values of param in a for-loop. I am using Python 3.6.9 on Linux. Thanks!

Answer 1

You can override config.param value, before mp.Pool() initiate like below.

# main.py
import multiprocessing as mp
import config, solver

config.param = 1 # trick is that

val_list = [None for i in range(3)]
pool = mp.Pool(10)
print('testing param: ', config.param)
results = pool.map(solver.solve, [val for val in val_list]) # multiprocessing
#results = [solver.solve(val) for val in val_list]          # basic