Multiprocessing with Joblib: Parallelising over one argument of a function

Question

Is there a way for a parallel function to take multiple arguments but only parallel over one of them ?

Say I've got some code:

def my_function(graph,graph_coefficients, thing_i_want_to_parallelise_over):

     <do_thing>

     return value 


results = Parallel(n_job2=2)(delayed(my_function(one_graph,graph_coefficients)(thing_i_want_to_parallelise_over) for thing_i_want_to_parallelise_over in range(1,3))

Is there a way to do this ? There are multiple functions to call so doing a simple wraparound function is not really an option.

Answer 1

I don't know if I understand your problem but you incorrectly formated it

Yous should create tuple with all arguments

(one_graph, graph_coefficients, x) for x in range(1,3)   # args

and then you should use it with

delayed( my_function )

like

results = Parallel(n_jobs=2)( 
                delayed( my_function )
                (one_graph, graph_coefficients, x) for x in range(1,3)
          )

---

Eventually you could try with lambda

lambda x: my_function(one_graph, graph_coefficients,x)

and then you could use

(x) for x in range(1,3)

like

results = Parallel(n_jobs=2)(
                delayed( lambda x: my_function(one_graph, graph_coefficients,x) ) 
                (x) for x in range(1,3) 
          )

---

Or with functools.partial

partial(my_function, one_graph, graph_coefficients)

like

from functools import partial

results = Parallel(n_jobs=2)(
                delayed( partial(my_function, one_graph, graph_coefficients) ) 
                (x) for x in range(1,3) 
          )

---

Minimal working code

from joblib import Parallel, delayed

def my_function(graph, graph_coefficients, thing_i_want_to_parallelise_over):
    print('my_function:', graph, graph_coefficients, thing_i_want_to_parallelise_over)
    value = 2 * thing_i_want_to_parallelise_over
    return value 

one_graph = 'A'
graph_coefficients = 'B'

# ----

results = Parallel(n_jobs=2)(
                delayed( my_function ) 
                (one_graph, graph_coefficients, x) for x in range(1,3) 
          )

print('results:', results)

# ----

results = Parallel(n_jobs=2)(
                delayed( lambda x: my_function(one_graph, graph_coefficients,x) ) 
                (x) for x in range(1,3) 
          )

print('results:', results)

# ----

from functools import partial

results = Parallel(n_jobs=2)(
                delayed( partial(my_function, one_graph, graph_coefficients) ) 
                (x) for x in range(1,3) 
          )

print('results:', results)