Search code examples
pythonparallel-processingscikit-learnjoblib

What is this joblib Parallel syntax doing? So many parentheses


Scikit-learn often uses joblib to parallelize with calls like Parallel(n_jobs=n_jobs)(delayed(function)(param_to_function) for param_to_function in iterable).

This helpful question and answer indicate that this double-parenthesis business means the second set is passed to whatever is returned by the call involving the first set, which makes a lot of sense if the thing returned is a callable.

But here the thing returned by Parallel(n_jobs=n_jobs) should be a Parallel object, right? And then we're passing it a generator object given by the loop in the second parenthetical. You shouldn't be able to directly pass a generator to a class after construction like that. There should be some function call between the object and the input. Or in python is there __some_special_function__ that works with this syntax?

What is this syntax doing, exactly?


Solution

  • The "special function" is probably just a __call__ method. An instance of a class with that method can be called just like a function. In this case, Parallel presumably defines __call__ to accept a generator.

    (Note, that's not to say it's a good idea to write code like your example. It's needlessly confusing.)