Search code examples
pythonmultithreadingconcurrent.futures

How to pass multiple arguments to concurrent.futures.ThreadPoolExecutor.map() function?


I'm writing a program to download list of image urls with multithreading. My function looks like following.

from wget import download

def down(url, path=None):
    """Download image urls.
    :param url: url of the image(mandatory)
    :param path: path to be downloaded images(optional) 
    """
    if path != None:    # if path is provided
        try:
            chdir(path) # change directory
            download(url)
        except ValueError:    # excepting urls that aren't images
            pass
    else:   # if path is not provided
        # again search for non-image urls and except them   
        try:
            download(url)
        except: 
            pass

I can download images with just passing url without path as follows.

images = [] # list of image urls

# multithreading downloading
with concurrent.futures.ThreadPoolExecutor() as executor:
    executor.map(down, images)

But when I try to pass path as below, It does nothing (doesn't download images into the path).

images = []
path = "/home/dead101/Downloads"

with concurrent.futures.ThreadPoolExecutor() as executor:
    executor.map(lambda p: down(*p), (images, path))

So my question is how to pass a function with multiple arguments to executor.map() function?


Solution

  • Use functools.partial:

    from functools import partial
    
    images = []
    path = "/home/dead101/Downloads"
    
    with concurrent.futures.ThreadPoolExecutor() as executor:
        executor.map(partial(down, path=path), images)