Search code examples
pythonpandasbinaryfiles

Storing Pandas dataframe in working memory


Is there some way to take a dataframe, say,

df = pd.DataFrame({'a':[1,2,3], 'b':[4,5,6]})

and store it in temp memory as a binary object that can then be opened with

open(df, 'rb')

So then, rather than do something like

open('/home/user/data.csv', 'rb')

the code would be

df = pd.DataFrame({'a':[1,2,3], 'b':[4,5,6]})

df_rb = *command to store in temp working memory as binary readable*

open(df_rb, 'rb')

Solution

  • Pandas has df.to_pickle() method:

    From the docs:

    Pickle (serialize) object to file.

    df.to_pickle("./dummy.pkl")
    

    Then read this pickled df using read_pickle()

    From the docs:

    Load pickled pandas object (or any object) from file.

    unpickled_df = pd.read_pickle("./dummy.pkl")