I'm not exactly adept when it comes to Python, but I've really put myself in a pickle. Puns aside, I'm using Tkinter, and I have a particular function that let's me save all of my global variables with ease:
def start_save(globals_=None) :
global quick_save_file, options
if globals_ is None:
globals_ = globals()
the_file = tkFileDialog.asksaveasfilename(**file_opt)
my_shelf = shelve.open(the_file,'c') # 'n' for new
for key, value in globals_.items():
if not key.startswith('__'):
try:
my_shelf[key] = globals()[key]
add_line( "Saved %r" % key)
except TypeError:
# __builtins__, my_shelf, and imported modules can not be shelved.
add_line(('ERROR shelving: {0}'.format(key)))
except :
add_line( "Cannot pickle %r" % key)
quick_save_file = the_file
options['initialfile'] = quick_save_file
my_shelf.close()
Basically we're looking at what comes after try: where it saves all of my global stuff into a file of the user's choosing. Now, I know Tkinter widgets can't be saved, but I accidently created a widget, and gridded it on the same line, like so:
ldrones_e = Label(frame_drones, text = "Number of drones:").grid(row = 3, column = 1, columnspan = 2)
This caused my shelving function to read it as None as opposed to a widget! So instead of saying it can't do anything with it, it saved it as None. Now, when I open said file, it takes my widget variable and sets it to none. I can't save over it, with .grid on a seperate line, because then it won't be saved. I can't use del to get rid of the variable, because it's in a different file. (Actually, I probably can, I'm just ignorant as to how to do that)
So my question is how do I remove that pesky ldrones_e variable from my pickle file so my program will use the new ldrones_e variable that doesn't have.grid attached to it.
If you absolutely had to, you could delete the relevant portions of the pickle by hand. This can be possible using pickletools.dis… however, you have to understand the recursive way that pickle works. Pickles are made by recursing into an object, and pickling it's state dependencies before the object itself can be pickled… and then similarly recursively for all of the dependencies… and so on, until that particular recursive branch hits an object that does not require dependencies to pickle. Then once all of the dependency objects are pickled, you finally are done.
Here you can see pickletools.dis showing what each portion of the pickle is:
>>> import pickletools
>>> import pickle
>>> pik = pickle.dumps(dict(zip(list('abcde'),[1,2,3,4,5])))
>>> pickletools.dis(pik)
0: ( MARK
1: d DICT (MARK at 0)
2: p PUT 0
5: S STRING 'a'
10: p PUT 1
13: I INT 1
16: s SETITEM
17: S STRING 'c'
22: p PUT 2
25: I INT 3
28: s SETITEM
29: S STRING 'b'
34: p PUT 3
37: I INT 2
40: s SETITEM
41: S STRING 'e'
46: p PUT 4
49: I INT 5
52: s SETITEM
53: S STRING 'd'
58: p PUT 5
61: I INT 4
64: s SETITEM
65: . STOP
highest protocol among opcodes = 0
>>> pik
"(dp0\nS'a'\np1\nI1\nsS'c'\np2\nI3\nsS'b'\np3\nI2\nsS'e'\np4\nI5\nsS'd'\np5\nI4\ns."
and here is dill, showing the route that an item takes to pickle. A printout like F1: … is the start of pickling an object, while # F1 is the end of pickling that object:
>>> import dill
>>> dill.detect.trace(True)
>>> dill.dumps(dict(zip(list('abcde'),[1,2,3,4,5])))
D2: <dict object at 0x10c5c9e88>
# D2
'\x80\x02}q\x00(U\x01aq\x01K\x01U\x01cq\x02K\x03U\x01bq\x03K\x02U\x01eq\x04K\x05U\x01dq\x05K\x04u.'
>>>
>>> def foo(x):
... def bar(y):
... return x+y
... return bar
...
>>> dill.dumps(foo)
F1: <function foo at 0x10c60a9b0>
F2: <function _create_function at 0x10c5a68c0>
# F2
Co: <code object foo at 0x10b6130b0, file "<stdin>", line 1>
F2: <function _unmarshal at 0x10c5a6758>
# F2
# Co
D1: <dict object at 0x10b51a168>
# D1
D2: <dict object at 0x10c5c4910>
# D2
# F1
'\x80\x02cdill.dill\n_create_function\nq\x00(cdill.dill\n_unmarshal\nq\x01U\xd6c\x01\x00\x00\x00\x02\x00\x00\x00\x02\x00\x00\x00\x03\x00\x00\x00s\x13\x00\x00\x00\x87\x00\x00f\x01\x00d\x01\x00\x86\x00\x00}\x01\x00|\x01\x00S(\x02\x00\x00\x00Nc\x01\x00\x00\x00\x01\x00\x00\x00\x02\x00\x00\x00\x13\x00\x00\x00s\x08\x00\x00\x00\x88\x00\x00|\x00\x00\x17S(\x01\x00\x00\x00N(\x00\x00\x00\x00(\x01\x00\x00\x00t\x01\x00\x00\x00y(\x01\x00\x00\x00t\x01\x00\x00\x00x(\x00\x00\x00\x00s\x07\x00\x00\x00<stdin>t\x03\x00\x00\x00bar\x02\x00\x00\x00s\x02\x00\x00\x00\x00\x01(\x00\x00\x00\x00(\x02\x00\x00\x00R\x01\x00\x00\x00R\x02\x00\x00\x00(\x00\x00\x00\x00(\x01\x00\x00\x00R\x01\x00\x00\x00s\x07\x00\x00\x00<stdin>t\x03\x00\x00\x00foo\x01\x00\x00\x00s\x04\x00\x00\x00\x00\x01\x0f\x02q\x02\x85q\x03Rq\x04c__builtin__\n__main__\nU\x03fooq\x05NN}q\x06tq\x07Rq\x08.'
>>>
>>> dill.detect.trace(False)
However, my best suggestion is if you plan on deleting objects from a pickle, then pickle in a more sensible way… lets's say you wanted to pickle a dict of objects (like everything in globals()). By using klepto, you can save each object in a dict as a pickled object to different file within a single directory.
>>> import klepto
>>> d = klepto.archives.dir_archive('saveme', serialized=True, cached=False)
>>> d.update(globals())
>>> d.keys()
['pickletools', 'dill', 'pik', 'd', '__builtins__', 'klepto', '__package__', '__name__', 'foo', 'pickle', '__doc__']
>>>
We quit, and start a new interpreter session:
Python 2.7.10 (default, May 25 2015, 13:16:30)
[GCC 4.2.1 Compatible Apple LLVM 5.1 (clang-503.0.40)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import klepto
>>> d = klepto.archives.dir_archive('saveme', serialized=True, cached=False)
>>> d.keys()
['pickletools', 'dill', 'pik', 'd', '__builtins__', 'klepto', '__package__', '__name__', 'foo', 'pickle', '__doc__']
>>> for i,j in d.items():
... globals()[i] = j
...
>>> foo(3)(4)
7
>>>
Each object is individually accessible from it's own file… so you can simply pop one of them. You can also use cached=True, and load whichever objects you like into memory (not shown) -- with cached=False, no objects are loaded into memory, and the file backends are interacted with directly.
>>> x = d.pop('d')
>>> del d['pik'], x
>>> d.keys()
['pickletools', 'dill', '__builtins__', 'klepto', '__package__', '__name__', 'foo', 'pickle', '__doc__']
>>> d['foo'](3)(4)
7