Search code examples
pythonpython-3.xmultithreadingmultiprocessingpython-multiprocessing

Parallel computing in Python Similar to MATLAB


I have been using parfor in MATLAB to run parallel for loops for quite some time. I need to do something similar in Python but I cannot find any simple solution. This is my code:

t = list(range(1,3,1))
G = list(range(0,3,2))
results = pandas.DataFrame(columns = ['tau', 'p_value','G','t_i'],index=range(0,len(G)*len(t)))
counter = 0  
for iteration_G in list(range(0,len(G))):
    for iteration_t in list(range(0,len(t))):
        
        matrix_1,matrix_2 = bunch of code


        tau, p_value = scipy.stats.kendalltau(matrix_1, matrix_2)
        results['tau'][counter] = tau
        results['p_value'][counter] = p_value
        results['G'][counter] = G[iteration_G]
        results['t_i'][counter] = G[iteration_t]
        counter = counter + 1

I would like to use the parfor equivalent in the first loop.


Solution

  • I found a solution using parfor. It is still a bit more complicated than MATLAB's parfor but it's pretty close to what I am used to.

    t = list(range(1,16,1))
    G = list(range(0,62,2))
    for iteration_t in list(range(0,len(t))):
        @parfor(list(range(0,len(G))))
        def fun(iteration_G):
            result = pandas.DataFrame(columns = ['tau', 'p_value'],index=range(0,1))
    
            matrix_1,matrix_2 = bunch of code   
    
    
    
            tau, p_value = scipy.stats.kendalltau(matrix_1, matrix_2)     
            result['tau'] = tau
            result['p_value'] = p_value
            fun = numpy.array([tau,p_value])
            return fun