Search code examples
pythonmultiprocessingpython-multiprocessing

Multiprocessing a function with several inputs


In Python the multiprocessing module can be used to run a function over a range of values in parallel. For example, this produces a list of the first 100000 evaluations of f.

def f(i):
    return i * i

def main():
    import multiprocessing
    pool = multiprocessing.Pool(2)
    ans = pool.map(f, range(100000))

    return ans

Can a similar thing be done when f takes multiple inputs but only one variable is varied? For example, how would you parallelize this:

def f(i, n):
    return i * i + 2*n

def main():
    ans = []
    for i in range(100000):
        ans.append(f(i, 20))

    return ans

Solution

  • There are several ways to do this. In the example given in the question, you could just define a wrapper function

    def g(i):
        return f(i, 20)
    

    and pass this wrapper to map(). A more general approach is to have a wrapper that takes a single tuple argument and unpacks the tuple to multiple arguments

    def g(tup):
        return f(*tup)
    

    or use a equivalent lambda expression: lambda tup: f(*tup).