Search code examples
rparallel-processingsnowfall

Using -snowfall- (or other packages) to parallel repeating a user-defined function

This question is related to this one, where I was asking how to replicate a user-defined function. Now I would like to parallelize the operations in order to save time. What I have preliminarly done is:

  1. I have defined a custom function my.fun(), which returns output, a matrix with 1000 rows and 20 columns.

  2. I replicate say 5 times output, and store the results in a single matrix called final through: final <- do.call(rbind, replicate(5, my.fun(), simplify=FALSE)). Hence, in this example final is a 5000-rows matrix.

What I would like to do now is to parallelize the 5 (or even more..) output replications before binding the results in the final matrix.

How would you do that? What I have (wrongly) done so far is:

    library(snowfall)

    sfInit(parallel = TRUE, cpus = 4, type = "SOCK")

    # previously defined objects manipulated within my.fun
    sfExport(...)

    my.fun = function() {
       ...
       return(output)
    }

    final <- do.call(rbind, sfSapply(1:5, fun=my.fun(), simplify=FALSE))

    sfStop()

but it returns: 

Error in get(as.character(FUN), mode = "function", envir = envir) : 
  object 'fun' of mode 'function' was not found

Any help would be greatly appreciated! Please, consider that I do not necessairly want to use -snowfall-: the final goal is to parallelize the computation of final in an efficient way (in reality I have to make a lot of replications..).

Solution

  • sfSapply expects fun to be a function, but you hand over the result of one call to my.fun. That is, you want to hand over my.fun, not my.fun ().