Search code examples
pythonmultithreadingargumentsthreadpoolpool

Pass different arguments to ThreadPool.apply_async() method in python


I want to pass different arguments to ThreadPool.apply_async() method. How I can do it? I thought it was done in the following way, but I was wrong. Could I use Thread instead? Thank you in advance

#!/usr/bin/env python
import multiprocessing

def testMethod(arg1, arg2, arg3):
    print(f"{arg1} - {arg2} - {arg3}"

if __name__ == '__main__':
    pool = multiprocessing.Pool()
    for t in range(1000):
        pool.apply_async(testMethod, t, arg2, arg3)

Solution

  • The type hints on the method signature in typeshed show, that postional arguments must be supplied as an iterable:

    def apply_async(self,
                    func: (...) -> _T,
                    args: Iterable = ...,
                    kwds: Mapping[str, Any] = ...,
                    callback: Optional[(_T) -> None] = ...,
                    error_callback: Optional[(BaseException) -> None] = ...)
    

    Your call to apply_async should look like this:

    pool.apply_async(testMethod, args=(t, 'foo', 'bar'))
    

    Or, using a keyword argument mapping:

    pool.apply_async(testMethod, kwds={'arg1': t, 'arg2': 'foo', 'arg3': 'bar'})
    

    As an aside: Your code example does not use a thread pool; multiprocessing.Pool is a class that represents a process pool, i.e. this creates seperate processes to act as workers.
    If you want to use a thread pool, go for:

    from multiprocessing.pool import ThreadPool
    pool = ThreadPool()
    

    The interface is similar to that of the process pool.