Can I fire & forget callables submitted to a thread pool?

Question

In my Python 3 application I have to deal with many small (simultaneous) I/O tasks that should not block the main thread, so I want to make use of thread pool:

from concurrent.futures import ThreadPoolExecutor

class MyApp:
    def __init__(self):
        self.app_thread_pool = ThreadPoolExecutor()

    def submit_task(self, task):
        self.app_thread_pool.submit(MyApp.task_runner, task)

    @staticmethod
    def task_runner(task):
        # Do some stuff with the task, save to disk, etc.

This works fine, the jobs are being submitted and started in the threads of the thread pool, the tasks do what they are supposed to do. Now, reading the documentation on the concurrent.futures module, it seems this module / thread pool is to be used with Future objects in order to handle the outcome of the submitted tasks. In my case, however, I am not interested in these results, I want to fire & forget my tasks, they are able to handle themselves.

So my question is: Do I have to use futures, or can I simply submit() a task, as shown above, and ignore any Future objects returned from the job submission? I ask in terms of memory and resouces management. Note that I also don't want to use the thread pool using the with statement, as described in the docs, because I need this thread pool over the whole lifetime of my application and the main thread starting the thread pool has many other things to do ...

Answer 1

No, of course you don't have to use the returned Future objects. As you have discovered, your code seems to correctly work without doing so. It's just a question of observability and robustness.

Keeping the Futures allows you to keep track of the submitted tasks. You can learn when they finish, retrieve their results, cancel them, etc. This is important for just knowing what's going on with your tasks.

But an arguably more important reason to keep tabs on the tasks is for robustness. What if one of the tasks fails? You probably want to retry it somehow. And note that there are very few tasks that cannot fail. You say your tasks are "I/O", which is a classic example of something where failures are common, and only known after you try something.

So while there is nothing forcing you to keep track of the futures, you probably should, especially if you need a robust, long-running application.