I have to use a nested for loop for a particular problem when the iterators are run and time for each run. I am trying to carry it out for different values of a hyperparameter(here T). I am trying to parallelise this process(3 processes) using multiprocessing.pool method. But I am not able to figure out how to implement it.
def simulate(T,runs,time,param1, param2, param3, param4):
for i in tqdm(range(runs)):
#Reset parameters
for j in range(time):
#Do some mathematics
#Some more mathematics
return (some output)
As it can be seen that the number of parameters for the function are many. So also I am not sure how to incorporate in functools.partial. Any guidelines?
If I understand you correctly, you want to run the simulate() method with different values of T and compare the results. To implement this using multiprocessing, you just need to set up a Pool with the right number of processes, and use map to run your function on a list of values of T. You also need to use partial to turn your function from one that takes seven arguments into a function that only needs one, with constant values for the other six. This is important because map needs to know which argument is the one that is being varied. Here's a (nontested) example:
import multiprocessing as mp
from functools import partial
# Create pool with desired number of processes
pool = mp.Pool( processes=3 )
# Make a partial function with preset values for params
partial_function = partial( simulate, runs=runs,
time=time, param1=param1, param2=param2,
param3=param3, param4=param4 )
# Dummy values for what T should be
options_for_T = [100, 200, 300, 400, 500, 600]
# Results will be a list of the output of simulate() for each T
results = pool.map( partial_function, options_for_T )
EDIT: I should also point out that using tqdm here might be counterproductive, since all your processes will be talking over each other