I've got a situation, when using the parallel function of joblib (v0.16.0) I have this lines on my code:
with parallel_backend('loky', n_jobs=8):
lineas = Parallel(verbose=10)(delayed(apply_prior_ind_def)(g) for g in df1_merge.groupby(['S1EMP','CONTRA1']))
The problem here is that sometimes the execution fails under the following message:
BrokenProcessPool: A task has failed to un-serialize. Please ensure that the arguments of the function are all picklable.
There is no apparent traceback to this issue, because without making any changes to the code and just restarting the terminal, it can sometimes restart the execution and finish it correctly.
Hope someone have had a similar issue when using parallel, and can shed a light on it.
Many thanks in advance.
Using Spyder3 and Python 3.6.5
What has worked for me is to have 2 terminals open at the same time. The first one is the one that fails to launch, but the second one is able to run the whole parallel code.