Search code examples
rmemoise

How to cache parallelly computed results with memoise::memoise?

I am precomputing results and caching them with the memoise function of the memoise package. Unfortunately, if I try to speed up the calculations by running them in parallel with parallel::mclapply, the memoisation does not happen.

> f <- memoise::memoise(function (a, b) a)
> memoise::has_cache(f)("foo", "bar")
[1] FALSE

> parallel::mclapply(c("bar", "baz"), f, a = "foo")
[[1]]
[1] "foo"

[[2]]
[1] "foo"

> memoise::has_cache(f)("foo", "bar")
[1] FALSE

> sapply(c("bar", "baz"), f, a = "foo")
  bar   baz 
"foo" "foo" 
> memoise::has_cache(f)("foo", "bar")
[1] TRUE

I suppose that the memoised function is copied and then discarded in each fork... Is there a simple alternative using other functions or packages ?

Solution

  • Turning my comment into an answer:

    The R.cache package (I'm the author) memoizes to file, which means the cache works across R sessions and in parallel R processes. Obviously, caching to file adds a bit of overhead compared to caching to memory.