Search code examples
rparallel-processingfunctional-programminglexical-scopelexical-closures

Parallellize Independent Function Calls that Each Modify Function's Parent Environment


I'd like to find a way to parallelize repeated independent function calls in which each call modifies the function's parent environment. Each execution of the function is independent, however, for various reasons I am unable to consider any other implementation that doesn't rely on modifying the function's parent environment. See simplified example below. Is there a way to pass a copy of the parent environment to each node? I am running this on a linux system.

 create_fun <- function(){

        helper <- function(x, params) {x+params}
        helper2 <- function(z) {z+helper(z)}

        master <- function(y, a){
            parent <- parent.env(environment())
            formals(parent[['helper']])$params <- a
            helper2(y)}

       return(master)
}

# function to be called repeatedly
master <- create_fun()

# data to be iterated over
x <- expand.grid(1:100, 1:5)

# vector where output should be stored
results <- vector("numeric", nrow(x))

# task I'd like to parallelize
for(i in 1:nrow(x)){
    results[i] <- master(x[i,1], x[i, 2])
}

Solution

  • Functions do maintain references to their parent environments. You can look at the contents of the environment of master (the environment created by create_fun)

    ls (environment(master) )
    # [1] "helper"  "helper2" "master" 
    

    Using %dopar% you could do

    ## Globals
    master <- create_fun()
    x <- expand.grid(1:100, 1:5)
    
    ## Previous results
    for(i in 1:nrow(x)){
        results[i] <- master(x[i,1], x[i, 2])
    }
    
    library(parallel)
    library(doParallel)
    cl <- makePSOCKcluster(4)
    registerDoParallel(cl)
    
    ## parallel
    res <- foreach(i=1:nrow(x), .combine = c) %dopar% {
        master(x[i,1], x[i,2])
    }
    
    all.equal(res, results)
    # TRUE