Search code examples
pythonpicklejoblib

joblib: cannot load dumped joblib file from another directory


I am trying to load a dumped joblib file from another directory, it's giving the following error

Traceback (most recent call last):
  File "D:\dir\loader.py", line 5, in <module>
    foob = joblib.load(r'D:\foob.joblib')
  File "C:\programming\programmeFiles\anaconda\envs\tensor_gpu\lib\site-packages\joblib\numpy_pickle.py", line 585, in load
    obj = _unpickle(fobj, filename, mmap_mode)
  File "C:\programming\programmeFiles\anaconda\envs\tensor_gpu\lib\site-packages\joblib\numpy_pickle.py", line 504, in _unpickle
    obj = unpickler.load()
  File "C:\programming\programmeFiles\anaconda\envs\tensor_gpu\lib\pickle.py", line 1088, in load
    dispatch[key[0]](self)
  File "C:\programming\programmeFiles\anaconda\envs\tensor_gpu\lib\pickle.py", line 1376, in load_global
    klass = self.find_class(module, name)
  File "C:\programming\programmeFiles\anaconda\envs\tensor_gpu\lib\pickle.py", line 1426, in find_class
    __import__(module, level=0)
ModuleNotFoundError: No module named 'foo'  

Minimal code to reproduce the error
The directory looks like this

D-
   \
    foo.py
    dumper.py
    foob.joblib
    dir-
        \
         loader.py 

foo.py

class Foo:
    def __init__(self):
        self.s = "here at Foo"
    def __repr__(self):
        return "fooObject"

dumper.py

import joblib
import foo

foob = foo.Foo()
joblib.dump(foob,'foob.joblib')

loader.py

import joblib
foob = joblib.load(r'D:\foob.joblib')
print(foob)  

If I try to load the foob.joblib from the same directory(D) it works fine, the error occurs while running loader.py from different directory


Solution

  • joblib.load() requires any modules required for unpickling to be in the path. So you need to add the location of foo.py the system path at loading time somehow.

    For instance add:

    import sys
    import os
    this_dir = os.path.dirname(__file__) # Path to loader.py
    sys.path.append(os.path.join(this_dir, <rel_path_to_foo.py>))
    

    to the top of loader.py