Is there some way to take a dataframe, say,
df = pd.DataFrame({'a':[1,2,3], 'b':[4,5,6]})
and store it in temp memory as a binary object that can then be opened with
open(df, 'rb')
So then, rather than do something like
open('/home/user/data.csv', 'rb')
the code would be
df = pd.DataFrame({'a':[1,2,3], 'b':[4,5,6]})
df_rb = *command to store in temp working memory as binary readable*
open(df_rb, 'rb')
Pandas has df.to_pickle()
method:
From the docs:
Pickle (serialize) object to file.
df.to_pickle("./dummy.pkl")
Then read this pickled df using read_pickle()
From the docs:
Load pickled pandas object (or any object) from file.
unpickled_df = pd.read_pickle("./dummy.pkl")