Python Multiprocessing TimeOut which sub function and inputs

Question

I have a process with a lot of sub functions runing with Multiprocessing pool.imap_unordered. Sometimes, the process may stuck. I currently managed that with a timeout as follow :

futures_res = pool.imap_unordered(ImageRequestedTypeGenerationWrapper, InputData.copy()) 

out1, out2, = futures_res.next(timeout=timeout * 60)

I would like to identify which sub function with which parameters failed. Could you advise a method ?

Answer 1

Try to wrap the call to the subfunction in a try-except block and to log any errors that occur. You can then check the logs to see which subfunction calls failed.

def process_subfunction(inputs):
    try:
        # Call subfunction with inputs
        result = subfunction(inputs)
    except Exception as e:
        # Log error and inputs
        logging.error(f"Error occurred while calling subfunction with inputs {inputs}: {e}")
        result = None
    return result

futures_res = pool.imap_unordered(process_subfunction, InputData.copy()) 

out1, out2, = futures_res.next(timeout=timeout * 60)

This will log any errors that occur when calling the subfunction, along with the inputs that were passed to the subfunction.