Search code examples
pythonmultiprocessingkeyword-argument

How to specify a certain keyword argument in Python multiprocessing


For a function with many arguments, including position and keyword ones. How to specify a list to a keyword argument without repeat previous keyword arguments?

E.g.

import multiprocessing
from functools import partial

def foo(a, b, c=1, d=2, e=3):
    print(multiprocessing.current_process().name)
    print(a + b + c + d + e)
    return a + b + c + d + e

with multiprocessing.Pool(processes=2) as pool:
    # I would like to give [1,2,3,4] to "e", something like
    # dic = {'e': [1,2,3,4]}
    # Yet we still have c = 1, d = 2.
    results = pool.map(partial(foo, 2, 3), dic)

The argument list can be long, and I don't want to type "c=1, d=2" in every partial function.

I know it's possible to use multiprocessing.apply, but I would like keep consistent with other usages of this function, and using map only.


Solution

  • I think you are going to need some kind of wrapper function because it seems that filling keyword arguments with partial can't be done. At the most general, you could have a caller function that takes a tuple of (function, args, kwargs) -- and pass to pool.map this generic caller and a list of such 3-element tuples:

    import multiprocessing
    from functools import partial
    
    def foo(a, b, c=1, d=2, e=3):
        print(multiprocessing.current_process().name)
        print(a, b, c, d, e, a + b + c + d + e)
        return a + b + c + d + e
    
    def generic_caller(tup):
        func, args, kwargs = tup
        return func(*args, **kwargs)
    
    with multiprocessing.Pool(processes=2) as pool:
    
        tups = [(foo, [2, 3], {'e': val})
                for val in [1, 2, 3, 4]]
    
        results = pool.map(generic_caller, tups)
        print(results)
    

    This gives the following (note that I added the individual variables to the print statement inside foo):

    ForkPoolWorker-2
    2 3 1 2 1 9
    ForkPoolWorker-2
    2 3 1 2 2 10
    ForkPoolWorker-2
    2 3 1 2 4 12
    ForkPoolWorker-1
    2 3 1 2 3 11
    [9, 10, 11, 12]
    

    Or you might prefer to hard-code func and/or args in the wrapper, and omit any hard-coded items from the tuples that are passed.

    Unfortunately solutions involving using a closure as the wrapper function (to avoid having to pass the same values of a and b every time) run into problems that this is not picklable, so would work with map but not pool.map.