Search code examples
pythonvariablesmultiprocessingglobalpool

Python pass variable to multiprocessing pool


I been trying to figure out this for ages, but wondering if anyone might know how I can pass the s variable to the pool without making it into an argument?

import ctypes
import multiprocessing as mp
import os

def worker1(n):
    k = n*3
    print(k)
    print(s)
    # print(ctypes_int.value)

if __name__ == '__main__':
    # global s
    somelist = [1,2,3]

    # ctypes_int = mp.Value(ctypes.c_wchar_p , "hi")
    s = "TESTING"
    # worker1(1)
    p = mp.Pool(1)
    p.map(worker1,somelist)

This is the error I am getting:

3
6
9
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "C:\Program Files\Python\Python37\lib\multiprocessing\pool.py", line 121, in worker
    result = (True, func(*args, **kwds))
  File "C:\Program Files\Python\Python37\lib\multiprocessing\pool.py", line 44, in mapstar
    return list(map(*args))
  File "C:\Users\light\AppData\Local\Temp\tempCodeRunnerFile.python", line 10, in worker1
    print(s)
NameError: name 's' is not defined
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\Users\light\AppData\Local\Temp\tempCodeRunnerFile.python", line 21, in <module>
    p.map(worker1,somelist)
  File "C:\Program Files\Python\Python37\lib\multiprocessing\pool.py", line 268, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "C:\Program Files\Python\Python37\lib\multiprocessing\pool.py", line 657, in get
    raise self._value
NameError: name 's' is not defined

Solution

  • You can pass your variable along with each item in somelist:

    import multiprocessing as mp
    
    def worker1(p):
        n,s = p
        k = n*3
        print(k)
        print(s)
    
    if __name__ == '__main__':
        somelist = [1,2,3]
    
        s = "TESTING"
        p = mp.Pool(1)
        p.map(worker1,[(n,s) for n in somelist])
    

    The parameter (n,s) gets passed as p and I unpack it into n,s.