Search code examples
pythonmultithreadingconcurrencykeyword-argument

Using concurrent.futures.ThreadPoolExecuter() with kwargs


How can I use concurrent.futures.ThreadPoolExecuter().map with a list that contains a dictionary of kwargs?

Using a func with only one argument, I can use the executer like this:

import concurrent.futures

def one_arg(arg):
    print(arg)

arg_list = [1, 2, 3]

with concurrent.futures.ThreadPoolExecutor() as executer:
    executer.map(one_arg, arg_list)

output:
1
2
3

Now, let's take a func with multiple input arguments:

def some_kwargs(kwarg_1, kwarg_2, kwarg_3):
    print('kwarg_1: {} - kwarg_2: {} - kwarg_3: {}'.format(kwarg_1, kwarg_2, kwarg_3))

kwargs = {"kwarg_1": "1_1", "kwarg_2": "1_2", "kwarg_3": "1_3"}
some_kwargs(**kwargs)

output:
'kwarg_1: 1_1 - kwarg_2: 1_2 - kwarg_3: 1_3'

Now, what I am really after is how to call some_kwargs using threading when I have a list of kwargs?

kwargs_list = [{"kwarg_1": "1_1", "kwarg_2": "1_2", "kwarg_3": "1_3"},
               {"kwarg_1": "2_1", "kwarg_2": "2_2", "kwarg_3": "2_3"},
               {"kwarg_1": "3_1", "kwarg_2": "3_2", "kwarg_3": "3_3"}]

with concurrent.futures.ThreadPoolExecutor() as executer:
    executer.map(some_kwargs, **kwargs_list)  # TypeError: map() argument after ** must be a mapping, not list

Solution

  • Use a lambda function to wrap some_kwargs:

    with concurrent.futures.ThreadPoolExecutor() as executer:
        executer.map(lambda x: some_kwargs(**x), kwargs_list)
    

    Note that **kwargs_list raises the above TypeError as kwargs_list is a list, not a dict.