Search code examples
pythonthreadpoolexecutor

Py3 ThreadPoolExecutor: how to get the numof remaining items until execution


I'm trying to add multiple long-running threads to ThreadPool in Python3. For each thread how can I determine how many tasks are in front of it before it gets executed?

My goal is to display to the user "You have X items pending before your task". It's not a problem if it's not precise when it reaches max_workers.

with futures.ThreadPoolExecutor(max_workers=2) as executor:
  future1 = executor.submit(task, 10) 
  future2 = executor.submit(task, 10)
  future3 = executor.submit(task, 10)

 # my naive failed attempt was
 numOfRemainingTasks = (len(executor)-1) - executor.indexof(future3)

Solution

  • If I understand correctly you want something along these lines?

    import concurrent.futures
    
    class CustomExecutor(concurrent.futures.ThreadPoolExecutor):
        def __init__(self, *args, **kwargs):
            super().__init__(*args, **kwargs)
        def submit(self, *args, **kwargs):
            print(f'There are {self._work_queue.qsize()} items in front of me')
            return super().submit(*args, **kwargs)
    
    def task(num):
        return num + 10
    
    
    with CustomExecutor(max_workers=2) as executor:
        futures = (executor.submit(task, 10) for _ in range(6))
        for future in concurrent.futures.as_completed(futures):
            print(future.result())
    
    There are 0 items in front of me
    There are 0 items in front of me
    There are 0 items in front of me
    There are 1 items in front of me
    There are 1 items in front of me
    There are 2 items in front of me
    ....