Search code examples
pythoncachingjoblib

joblib: load all cached values (or search through all of them)

Using joblib.Memory I can cache the results of some function, but I wonder if there is an efficient way to figure out the list of arguments for which there is a cached value.

For example, suppose that the function slow_func was computed and cached for some values of x, can I find out for which values I have cache?

from joblib import Memory
mem = Memory(location='cache')

@mem.cache
def slow_func(x):
    return x
Solution

  • Please use the following answer with caution: it uses non-public API of joblib.

    from joblib import Memory
mem = Memory(location='cache')

...

def iterate_cache(mem):
    """Return the list of inputs and outputs from `mem` (joblib.Memory cache)."""
    for item in mem.store_backend.get_items():
        path_to_item = os.path.split(os.path.relpath(item.path, start=mem.store_backend.location))
        result = mem.store_backend.load_item(path_to_item)
        input_args = mem.store_backend.get_metadata(path_to_item).get("input_args")
        yield input_args, result

print(list(iterate_cache(mem)))

    StoreBackend encapsulates all the methods related to cache storage. StoreBackend.load_item loads the cached results, and StoreBackend.load_metadata loads all the metadata, including input args.