I'm getting a memory error when trying to pickle a large numpy array for a deep learning problem shape: (7451, 1500, 1500, 1)). That stated, I see a few posts on klepto and read the docs but I'm not sure how to actually use klepto to save as a pickle file.
Can anyone break it down to a fifth grade level for me?
This is throwing the memory error:
pickle_out = open("X.pickle", "wb")
pickle.dumps(X, pickle_out)
pickle_out.close()
When I faced similar problem I could solve it using joblib. You would need first to have sklearn library installed, which can be done for example with
pip install sklearn
That is just basic idea, to better know how to install it go to https://scikit-learn.org/stable/install.html So, everything is pretty plane and is illustrated in following code
from sklearn.externals import joblib
import numpy as np
array=np.array([0,1,2]) # thats explanatory array, you should use your instead
filename = 'array.sav'
joblib.dump(array, filename)
Then, to load back your data when u need to use it:
array = load(filename, mmap_mode='r')