concurrent.futures.ThreadPoolExecutor swallowing exceptions (Python 3.6)

Question

I'm trying to use ThreadPoolExecutor in Python 3.6 on Windows 7 and it seems that the exceptions are silently ignored or stop program execution. Example code:

#!/usr/bin/env python3

from time import sleep

from concurrent.futures import ThreadPoolExecutor

EXECUTOR = ThreadPoolExecutor(2)


def run_jobs():
    EXECUTOR.submit(some_long_task1)
    EXECUTOR.submit(some_long_task2, 'hello', 123)
    return 'Two jobs was launched in background!'


def some_long_task1():
    print("Task #1 started!")
    for i in range(10000000):
        j = i + 1
    1/0
    print("Task #1 is done!")


def some_long_task2(arg1, arg2):
    print("Task #2 started with args: %s %s!" % (arg1, arg2))
    for i in range(10000000):
        j = i + 1
    print("Task #2 is done!")


if __name__ == '__main__':
    run_jobs()
    while True:
        sleep(1)

The output:

Task #1 started!
Task #2 started with args: hello 123!
Task #2 is done!

It's hanging there until I kill it with Ctrl+C.

However, when I remove 1/0 from some_long_task1, Task #1 completes without problem:

Task #1 started!
Task #2 started with args: hello 123!
Task #1 is done!
Task #2 is done!

I need to capture the exceptions raised in functions running in ThreadPoolExecutor somehow.

Python 3.6 (Minconda), Windows 7 x64.

Answer 1

ThreadPoolExecutor.submit returns a future object that represents the result of the computation, once it becomes available. In order to not ignore exceptions raised by the submitted function, you need to actually access this result. First, you can change run_job to return the created futures:

def run_jobs():
    fut1 = EXECUTOR.submit(some_long_task1)
    fut2 = EXECUTOR.submit(some_long_task2, 'hello', 123)
    return fut1, fut2

Then, have the top-level code wait for the futures to complete, and access their results:

import concurrent.futures

if __name__ == '__main__':
    futures = run_jobs()
    concurrent.futures.wait(futures)
    for fut in futures:
        print(fut.result())

Calling result() on a future whose execution raised an exception will propagate the exception to the caller. In this case the ZeroDivisionError will get raised at top-level.

Of course, instead of allowing the exception to be raised and propagated, you can place the result() invocation in a try block and catch to catch the exception. Alternatively, you can call the exception() method on the Future object to just retrieve the raised exception, if any.