How to catch Joblib/Parallel Exceptions in python?

Question

I have a use case for joblib's Parallel, delayed. I have included some feature that terminates a worker under certain conditions. However, when I do that, I am randomly yielding JoblibWebdriverException, Multiprocessing exception, JoblibURLerror, or just error.

To my great amusement, I don't find any section on how to (define?)/catch Exceptions in the docs.

When I do:

    try:
       Parallel(delayed(function))
    except (JoblibWebdriverException | error | 'Multiprocessing exception'): 
    # written with separate excepts in original code
       log_errors()

I yield name JoblibWebdriverException not defined followed by:

---------
Sub-process traceback
---------
Multiprocessing exception:
(trace stack)

How to catch undefined joblib Exceptions when using Parallel in python?

Answer 1

I would recommend using concurrent.Futures which has robust support for Exception handling. joblib is notorious for not being able to raise Exceptions from child processes to the main thread due to way the multi-processing is set up.