Passing a function that takes **kwargs into concurrent.futures.ThreadPoolExecutor.map

Question

I've just begun learning concurrent programming in Python.

I created a function which takes one positional argument and then **kwargs.

Code:

def download_from_api(api_client_method, **kwargs):
    try:
        response = api_client_method(**kwargs)
        return response
    except api.exceptions.APIException as e:
        print(f'{kwargs}: {api_client_method.__name__} - {e})

I'd like to pass this function to concurrent.futures.ThreadPoolExecutor().map() object.

Below is the code i used for trying, passing everything as positional args, but it didnt work:

these_should_be_passed_as_keyword_args = ['A', 'B']
n = len(these_should_be_passed_as_keyword_args)
functions = [api_client.relevant_method] * n

with concurrent.futures.ThreadPoolExecutor() as executor:
    results = executor.map(download_from_api, functions, these_should_be_passed_as_keyword_args)

Any help is really appreciated.

Thank you!

Answer 1

There are some good examples here: Pass multiple parameters to concurrent.futures.Executor.map?

Alternatively, I prefer using .submit() Here are the docs, and here is a tutorial

As I cant execute your download_from_api function, here is a basic example with a function that just prints random generated kwargs:

import string
import random
import concurrent.futures as cf


def your_function(argument, **kwargs):
    print(argument, kwargs)


random_kwargs = [{
    ''.join(random.choice(string.ascii_lowercase) for _ in range(4)):
        ''.join(random.choice(string.ascii_lowercase) for _ in range(4))
} for _ in range(10)]

with cf.ThreadPoolExecutor() as executor:
    futures = [executor.submit(your_function, argument, **kwargs) 
               for argument, kwargs in enumerate(random_kwargs)]