Search code examples
pythonpickledill

Loading pickled (dill) file containing dictionary of functions


This is a follow-up to a question I previously asked: Loading functions in pickle file that uses class wrapper, where I wanted to pickle a dictionary of function under class wrappers, but the class wrapper module couldn't load.

I have since solved this using dill instead of pickle:

func_dict = lowess_record()

wanted = ['func_dict', 'Interp1dPicklable', 'PolyValPicklable', 'dill', '__builtins__', 'wanted']

for name in globals().keys():
    if name not in wanted:
        del globals()[name]

del globals().wanted

with open('./func_dict.p', 'wb') as f:
    dill.dump(globals(), f)

I removed some unnecessary variables (involved in creating func_dict from data) from globals(), they weren't needed as I only wanted to work with func_dict.

Now when I load the pickled file using:

func_dict_loaded = dill.load(open("./func_dict.p", "rb"))

I get:

<In> func_dict_loaded.keys()

<Out> ['_dh', '__', 'dill', '_15', '__builtin__', '_i32', '_30', '_16', '_i15', quit', '_34', '_i11', '_i9', '_i8', '_i7', '_i6', '_i5', '_i4', '_i3', '_i2', _i1', '__package__', 'exit', 'get_ipython', '_i', '_i29', '_i26', '_i17', _i24', _i14', '_i22', '__doc__', '_i20', '_i16', '_i21', '_18', '_11', '_i34', __builtins__', '_ih', '_i28', 'sys', '_20', '_i27', '__name__', '___', '_i33', _', '_sh', '_i25', '_29', '_32', '_22', 'func_dict_loaded', '_i23', '_i13', _i12', '_iii', '_i10', '_13', '_12', '_ii', 'In', '_i31', '_i30', '_i19', _i18', _i35', '_oh', 'Out']

With no way to access the dictionary of functions! What do I need to do to get the intended output?


Solution

  • I seem to have fixed it, here's how:

    when dumping the function dictionary, instead of:

    dill.dump(globals(), f)
    

    I have used:

    dill.settings['recurse'] = True
    
    with open('./Data/function_dict.p', 'wb') as f:
        dill.dump([dill, interp1d, Interp1dPicklable, PolyValPicklable, func_dict], f)
    f.close()
    

    The first line is a tip I found here

    Then, when I load the .p file I call:

    dill.settings['recurse'] = True
    
    func_dict_loaded = dill.load(open("./Data/function_dict.p", "rb"))
    

    (Not sure if the first line is needed there but it doesn't seem to do any harm.)

    Then:

    func_dict_loaded[-1]
    

    Returns func_dict (note order of list when dumping).

    I can then use

    func_dict_loaded[-1]['func1'](x)
    

    to return the result of func_1 for variable x.

    Thanks to everyone who looked and commented/answered, I hope this can help someone else with a similar problem in the future!