Search code examples
pythonmultithreadingpython-multithreading

ThreadPoolExecutor: how to limit the queue maxsize?

I am using ThreadPoolExecutor class from the concurrent.futures package

def some_func(arg):
    # does some heavy lifting
    # outputs some results

from concurrent.futures import ThreadPoolExecutor

with ThreadPoolExecutor(max_workers=1) as executor:
    for arg in range(10000000):
        future = executor.submit(some_func, arg)

but I need to limit the queue size somehow, as I don't want millions of futures to be created at once, is there a simple way to do it or should I stick to queue.Queue and threading package to accomplish this?

Solution 
  • from concurrent.futures import ThreadPoolExecutor, wait, FIRST_COMPLETED

limit = 10

futures = set()

with ThreadPoolExecutor(max_workers=1) as executor:
    for arg in range(10000000):
        if len(futures) >= limit:
            completed, futures = wait(futures, return_when=FIRST_COMPLETED)
        futures.add(executor.submit(some_func, arg))