Getting original line number for exception in concurrent.futures

Question

Example of using concurrent.futures (backport for 2.7):

import concurrent.futures  # line 01
def f(x):  # line 02
    return x * x  # line 03
data = [1, 2, 3, None, 5]  # line 04
with concurrent.futures.ThreadPoolExecutor(len(data)) as executor:  # line 05
    futures = [executor.submit(f, n) for n in data]  # line 06
    for future in futures:  # line 07
        print(future.result())  # line 08

Output:

1
4
9
Traceback (most recent call last):
  File "C:\test.py", line 8, in <module>
    print future.result()  # line 08
  File "C:\dev\Python27\lib\site-packages\futures-2.1.4-py2.7.egg\concurrent\futures\_base.py", line 397, in result
    return self.__get_result()
  File "C:\dev\Python27\lib\site-packages\futures-2.1.4-py2.7.egg\concurrent\futures\_base.py", line 356, in __get_result
    raise self._exception
TypeError: unsupported operand type(s) for *: 'NoneType' and 'NoneType'

String "...\_base.py", line 356, in __get_result" is not endpoint I expected to see. Is it possible to get real line where exception was thrown? Something like:

  File "C:\test.py", line 3, in f
    return x * x  # line 03

Python3 seems to show correct line number in this case. Why can't python2.7? And is there any workaround?

Answer 1

I think the original exception traceback gets lost in the ThreadPoolExecutor code. It stores the exception and then re-raises it later. Here is one solution. You can use the traceback module to store the original exception message and traceback from your function f into a string. Then raise an exception with this error message, which now contains the line number etc of f. The code that runs f can be wrapped in a try...except block, which catches the exception raised from ThreadPoolExecutor, and prints the message, which contains the original traceback.

The code below works for me. I think this solution is a little hacky, and would prefer to be able to recover the original traceback, but I'm not sure if that is possible.

import concurrent.futures
import sys,traceback


def f(x):
    try:
        return x * x
    except Exception, e:
        tracebackString = traceback.format_exc(e)
        raise StandardError, "\n\nError occurred. Original traceback is\n%s\n" %(tracebackString)



data = [1, 2, 3, None, 5]  # line 10

with concurrent.futures.ThreadPoolExecutor(len(data)) as executor:  # line 12
    try:
        futures = [executor.submit(f, n) for n in data]  # line 13
        for future in futures:  # line 14
           print(future.result())  # line 15
    except StandardError, e:
        print "\n"
        print e.message
        print "\n"

This gives the following output in python2.7:

1
4
9




Error occurred. Original traceback is
Traceback (most recent call last):
File "thread.py", line 8, in f
   return x * x
TypeError: unsupported operand type(s) for *: 'NoneType' and 'NoneType'

The reason your original code gives the right location when run in Python 3 and not 2.7 is that in Python 3 exceptions carry the traceback as an attribute, and when re-raising an exception, the traceback is extended rather than replaced. The example below illustrates this:

def A():
    raise BaseException("Fish")

def B():
    try:
        A()
    except BaseException as e:
        raise e

B()

I ran this in python 2.7 and python 3.1. In 2.7 the output is as follows:

Traceback (most recent call last):
  File "exceptions.py", line 11, in <module>
    B()
  File "exceptions.py", line 9, in B
    raise e
BaseException: Fish

i.e. the fact that the exception was originally thrown from A is not recorded in the eventual output. When I run with python 3.1 I get this:

Traceback (most recent call last):
  File "exceptions.py", line 11, in <module>
    B()
  File "exceptions.py", line 9, in B
    raise e
  File "exceptions.py", line 7, in B
    A()
  File "exceptions.py", line 3, in A
    raise BaseException("Fish")
BaseException: Fish

which is better. If I replace raise e with just raise in the except block in B, then python2.7 gives the complete traceback. My guess is that when back-porting this module for python2.7 the differences in exception propagating were overlooked.