Search code examples
pythonpackagepandasmemoization

Pandas memoization

I have lengthy computations which I repeat many times. Therefore, I would like to use memoization (packages such as jug and joblib), in concert with Pandas. The problem is whether the package would memoize well Pandas DataFrames as method arguments.

Has anyone tried it? Is there any other recommended package/way to do this?

Solution

  • Author of jug here: jug works fine. I just tried the following and it works:

    from jug import TaskGenerator
import pandas as pd
import numpy as np


@TaskGenerator
def gendata():
    return pd.DataFrame(np.arange(343440).reshape((10,-1)))

@TaskGenerator
def compute(x):
    return x.mean()

y = compute(gendata())

    It is not as efficient as it could be as it just uses pickle internally for the DataFrame (although it compresses it on the fly, so it is not horrible in terms of memory use; just slower than it could be).

    I would be open to a change which saves these as a special case as jug currently does for numpy arrays: https://github.com/luispedro/jug/blob/master/jug/backends/file_store.py#L102