pickle error when serialize cython class with lambda

I use pickle and dill for follow lambda function and work fine :

import dill
import pickle

f = lambda x,y: x+y
s = pickle.dumps(f)

or even when used in class, for example:

file

foo.py

class Foo(object):
    def __init__(self):
        self.f = lambda x, y: x+y

file

test.py

import dill
import pickle
from foo import Foo

f = Foo()
s = pickle.dumps(f)  # or  s = dill.dumps(f)

but when build same file with format .pyx (foo.pyx) using cython, can't serialize with dill, pickle or cpickle, get this error :

Traceback (most recent call last): File "/home/amin/anaconda2/envs/rllab2/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 2878, in run_cod exec(code_obj, self.user_global_ns, self.user_ns) File "", line 1, in a = pickle.dumps(c) File "/home/amin/anaconda2/envs/rllab2/lib/python2.7/pickle.py", line 1380, in dumps Pickler(file, protocol).dump(obj) File "/home/amin/anaconda2/envs/rllab2/lib/python2.7/pickle.py", line 224, in dump self.save(obj) File "/home/amin/anaconda2/envs/rllab2/lib/python2.7/pickle.py", line 331, in save self.save_reduce(obj=obj, *rv) File "/home/amin/anaconda2/envs/rllab2/lib/python2.7/pickle.py", line 425, in save_reduce save(state) File "/home/amin/anaconda2/envs/rllab2/lib/python2.7/pickle.py", line 286, in save f(self, obj) # Call unbound method with explicit self File "/home/amin/anaconda2/envs/rllab2/lib/python2.7/site-packages/dill/_dill.py", line 912, in save_module_dict StockPickler.save_dict(pickler, obj) File "/home/amin/anaconda2/envs/rllab2/lib/python2.7/pickle.py", line 655, in save_dict self._batch_setitems(obj.iteritems()) File "/home/amin/anaconda2/envs/rllab2/lib/python2.7/pickle.py", line 669, in _batch_setitems save(v) File "/home/amin/anaconda2/envs/rllab2/lib/python2.7/pickle.py", line 317, in save self.save_global(obj, rv) File "/home/amin/anaconda2/envs/rllab2/lib/python2.7/pickle.py", line 754, in save_global (obj, module, name)) PicklingError: Can't pickle . at 0x7f9ab1ff07d0>: it's not found as foo.lambda

setup.py file for build cython

setup.py

from distutils.core import setup
from Cython.Build import cythonize

setup(ext_modules=cythonize("foo.pyx"))

then run in terminal:

python setup.py build_ext --inplace

Is there a way ?

Solution

I'm the dill author. Expanding on what @DavidW says in the comments -- I believe there are (currently) no known serializers that can pickle cython lambdas, or the vast majority of cython code. Indeed, it is much more difficult for python serializers to be able to pickle objects with C-extensions unless the authors of the C-extension code specifically build serialization instructions (as did numpy and pandas). In that vein... instead of a lambda, you could build a class with a __call__ method, so it acts like a function... and then add one or more of the pickle methods (__reduce__, __getstate__, __setstate__, or something similar)... and then you should be able to pickle instances of your class. It's a bit of work, but since this path has been used to pickle classes written in C++ -- I believe you should be able to get it to work for cython-built classes.