Search code examples
pythonserializationpicklelru

pickling lru_cached function on object

As part of parallellizing some existing code (with multiprocessing), I run into the situation that something similar to the class below needs to be pickled.

Starting from:

import pickle
from functools import lru_cache

class Test:
    def __init__(self):
        self.func = lru_cache(maxsize=None)(self._inner_func)

    def _inner_func(self, x):
        # In reality this will be slow-running
        return x

calling

t = Test()
pickle.dumps(t)

returns

_pickle.PicklingError: Can't pickle <functools._lru_cache_wrapper object at 0x00000190454A7AC8>: it's not the same object as __main__.Test._inner_func

which I don't really understand. By the way, I also tried a variation where the name of _inner_func was func as well, that didn't change things.

Solution

  • As detailled in the comments, the pickle module has issues when dealing with decorators. See this question for more details:

    Pickle and decorated classes (PicklingError: not the same object)