A question for using python "joblib.Parallel" function

Question

I want to repeat my function "Function_test" 100 times with Parallel for loop in python by using "joblib.Parallel", and all the results from 100 times for further analysis.

Here is my code:

import joblib

def FOMs_test(jj,image, Center_xy, BkgCenter_xy, roiSize_xy, Chnl,Resampling, Validation):

   # ... omit some code here

   _,dp[jj,ii], AUC[jj,ii], Pc[jj,ii] = Function_test (sig,bkg,channel,Noise,Resampling)
   print(jj)

return dp, AUC, Pc


dp, AUC, Pc = joblib.Parallel(n_jobs=3)(joblib.delayed(FOMs_test)(jj,image, Center_xy, BkgCenter_xy, roiSize_xy, Chnl,Resampling, Validation) for jj in range(2))

After I run the above code, I got the following error message:

0
1
0
1
0
1

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-2-81843d48c033> in <module>
----> 1 dp, AUC, Pc = joblib.Parallel(n_jobs=3)(joblib.delayed(FOMs_test)(jj,image, Center_xy, BkgCenter_xy, roiSize_xy, Chnl,Resampling, Validation) for jj in range(2))

ValueError: not enough values to unpack (expected 3, got 2)

How can I solve this?

Answer 1

The error message indicates that there are not enough values being returned from the joblib.Parallel call to unpack into the variables dp, AUC, and Pc.

This is because the joblib.Parallel call is currently only returning two sets of values, one for jj=0 and another for jj=1. To get the results of all 100 function calls, you need to adjust the range of the loop to range(100) instead of range(2).

Also, you need to use a container to store the results from each iteration of the loop, such as a list, and then concatenate the results from all iterations into the final variables dp, AUC, and Pc.

Here's an updated version of the code that should work:

import joblib

def FOMs_test(jj, image, Center_xy, BkgCenter_xy, roiSize_xy, Chnl, Resampling, Validation):
   # ... omit some code here
   _, dp_jj, AUC_jj, Pc_jj = Function_test(sig, bkg, channel, Noise, Resampling)
   print(jj)
   return dp_jj, AUC_jj, Pc_jj


results = joblib.Parallel(n_jobs=3)(
    joblib.delayed(FOMs_test)(jj, image, Center_xy, BkgCenter_xy, roiSize_xy, Chnl, Resampling, Validation)
    for jj in range(100)
)

# concatenate the results from all iterations
dp, AUC, Pc = map(lambda x: np.concatenate(x, axis=0), zip(*results))