Why does `joblib.delayed` not work as decorator?

Question

The package joblib has a function delayed which captures the arguments passed to the function. It can be used for example like this:

from joblib import delayed

def f(n):
    return n**2

delayed(f)(2)
# (<function f at 0x7f939eb3fe60>, (2,), {})
h = delayed(f)
h(2)
# (<function f at 0x7f939eb3fe60>, (2,), {})

The latter usage makes me think this could be used like any other decorator. This is supported by its documentation:

joblib.delayed(function, check_pickle=True)

Decorator used to capture the arguments of a function.

However, this fails:

from joblib import delayed

@delayed
def g(n):
    return n**2

# Traceback (most recent call last):
#  File "<stdin>", line 1, in <module>
#  File "/usr/lib/python2.7/dist-packages/joblib/parallel.py", line 158, in delayed
#    pickle.dumps(function)
#  File "/usr/lib/python2.7/copy_reg.py", line 70, in _reduce_ex
#    raise TypeError, "can't pickle %s objects" % base.__name__
#TypeError: can't pickle function objects

I tried this with Python 2.7.12

Answer 1

It looks like it's a namespace issue, which causes joblib.delayed to attempt to pickle the output function (instead of the original function).

I haven't looked into the joblib code, but the thing you proposed would be the way to do it:

import joblib

def _func(*args, **kwargs):
    'your code here'

func = joblib.delayed(_func)

It's not ideal, but it works.