Search code examples
pythonasynchronousparallel-processingpython-asynciopython-multithreading

python asynchronous map function that apply a function to every element of a list

I have a list of elements in python that I want to apply an API call to. The API involves web requests and I want to apply it synchronously. Since it is a third party API I can't really modify the code inside as well. I want to have something like this:

from API import get_user_handle
arr = [1234, 2345, 6789] # list of user id
new_arr = async_foreach(arr, get_user_handle)
print(new_arr) # e.g. ['Tom', 'Amy', 'Jerry']

This would be easy in nodejs by using the array map function. How do I build something similar in python?

Solution

  • Use concurrent.futures, I suggest using threads in this case, if you are IO bound:

    import concurrent.futures

with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor:
    new_arr = executor.map(get_user_handle, arr)