Search code examples
pythonpython-multiprocessingconcurrent.futures

How to make `concurrent.futures.ProcessPoolExecutor().map` work with kwonly args?


How to make concurrent.futures.ProcessPoolExecutor().map work with kwonly args? Is this even possible?

With positional args:


def worker_function(x):
    # Return the square of the passed argument:
    return x ** 2

# concurrent.futures example:
from concurrent.futures import ProcessPoolExecutor
with ProcessPoolExecutor() as executor:
    squares = list(executor.map(worker_function, (1, 2, 3, 4, 5, 6)))

How to make this work for a function accepting keyword only args:

def worker_function(*, x):
    # Return the square of the passed argument:
    return x ** 2

I tried this

with ProcessPoolExecutor() as executor:
    squares = list(executor.map(worker_function, (dict(x=x) for x in (1, 2))))

which fails because dict is not passed as kwargs for some reason

TypeError: worker_function() takes 0 positional arguments but 1 was given

Solution

  • okay, something like this seems to work:

    with ProcessPoolExecutor() as executor:
        # Using lambda, unpacks the dict (**f) into hello(**kwargs)
        kwargs_names = (dict(x=x, y=2) for x in range(4))
        executor.map(lambda f: worker_function(**f), kwargs_names)
    
    print(squares)
    

    ref